2022-04-02T02:59:27.780 INFO:root:teuthology version: 1.1.0-d877b023 2022-04-02T02:59:27.782 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --description rados/rook/smoke/{0-distro/ubuntu_20.04 0-kubeadm 0-nvme-loop 1-rook 2-workload/none 3-final cluster/3-node k8s/1.21 net/flannel rook/master} --archive /home/teuthworker/archive/yuriw-2022-04-01_18:43:08-rados-wip-yuri11-testing-2022-04-01-0842-distro-default-smithi/6773474 --owner scheduled_yuriw@teuthology --name yuriw-2022-04-01_18:43:08-rados-wip-yuri11-testing-2022-04-01-0842-distro-default-smithi -- /home/teuthworker/archive/yuriw-2022-04-01_18:43:08-rados-wip-yuri11-testing-2022-04-01-0842-distro-default-smithi/6773474/orig.config.yaml 2022-04-02T02:59:27.815 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2022-04-02T02:59:27.878 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2022-04-01_18:43:08-rados-wip-yuri11-testing-2022-04-01-0842-distro-default-smithi/6773474 branch: wip-yuri11-testing-2022-04-01-0842 description: rados/rook/smoke/{0-distro/ubuntu_20.04 0-kubeadm 0-nvme-loop 1-rook 2-workload/none 3-final cluster/3-node k8s/1.21 net/flannel rook/master} email: ceph-qa@ceph.io first_in_suite: false job_id: '6773474' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: yuriw-2022-04-01_18:43:08-rados-wip-yuri11-testing-2022-04-01-0842-distro-default-smithi nuke-on-error: true os_type: ubuntu os_version: '20.04' overrides: admin_socket: branch: wip-yuri11-testing-2022-04-01-0842 ceph: conf: mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: flavor: default sha1: 76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a kernel: hwe: true kubeadm: pod_network: flannel version: '1.21' rook: rook_image: rook/ceph:master workunit: branch: wip-yuri11-testing-2022-04-01-0842 sha1: 76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a owner: scheduled_yuriw@teuthology priority: 56 repo: https://github.com/ceph/ceph-ci.git roles: - - host.a - client.a - - host.b - client.b - - host.c - client.c sha1: 76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a sleep_before_teardown: 0 suite: rados suite_branch: wip-yuri11-testing-2022-04-01-0842 suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a targets: smithi029.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDXx1K5EJbC1SnSrQhDpp0xYGF3zZUQztC0UPOZYA4S8hMLwu7VVegXKf46H/YTknoskB+n4E2C/JilbCZJX97SuIUromwSyM19TOh1Oq17cqcxoIRlG9K/AIMNhvpKwr9jOBEaaJgRa8QvlR+yb9cykGBYZ2ac25/iAxTeQ+i5RhjjZ/bw/mzjnPUTh9IZAYFLknkrgt3f0BZtPtBOYCvEXLRK+bFW242iB2YGGe5Q5hqdLRtLvrmK+VAvDiqIDWJYMDA+jPwphgdLtZWkrqoyclKDZDwV6uIDuqf3UZOMBWMXeXADY7oRZAE64n57cMU17Q68DWytakR3DFsqE6Ho6ccww7fr8vE/CxcpHL8PBcYHHzcCZyF40Vt/MVy0S4VL24UOPgNr2Pv4T+J/M0Ryq7gyTpWj9wVGyNqy20wJpHejz/6QE5XhUxQuZqRggGZeEcCprHrBIoRJBjWZUiwaTyRWqsqoPFKh2d4boadHHLp3S83z6YnK/lU2lRjspC8= smithi104.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDQIwd2N4HDAEpCrUkfz8Umb9uREjNICB7DBxMNEl9cnTLtg5is51TnJVO8iygQ/dhNvcxBk2OTIyY60a6MyMjZlAxnBZ2hatstm5H4cyrR1u5E4R0AqKfld2oAXcy2Re+aTYHWsVwa6gTMl35WsGqkIIOzwDWzyEy/vEfFo72meCOrD7sd3o9loTeVn84PzbUAPCiKiet1070I5145Vs/CDb7VLyqXBeHZhe2ORiLQ8VmsdDoQu1MjgEUSn2AaU9Ov/43F898/ILam9lZGmCkcPs3LWtIV6F5tqPvo30CaSnHgIpm3srVaC+tqufsqH9VLZbbZasoneOFk2BFIGE85mVwgGvcX+iHN4Dm1sG7EYEuN0549bEFktZWQ4mWLhADG2LzXsAXq1aTs6AdI0jyEqgW8HzQQ7eedhirCJT5LZST5+ubSkjkS+posn78iFUJw53Q89TPz7k/08EFGEH3NziSMiq/d2K1e6zK3EOLFVNyaJKz76lvNPCkS9GAeZjE= smithi142.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC9t1B3iuoYVUmdppJ69Fc3Znv16edYPlsX9LTJJrWe+a2cSoCaba30tKCZzM8+dTUx8aIjpBj4//fkIZOX04AyF/dzE9Y73OD54B6h71QDg5VFu14naJtIYLi+HBr1hYBReEDerlBq05zUyRzd6t09oygilz2wcB1xJP3r7OgRdRhZfvEBt2+nBvnLjIL+DDEQIs+XIs4Pf0x7KoprVwBCMdvGKl76qhpj70NBQSyJvruaUHlrydQyqvXVORDE7Pn1PpYjKatC8i5UVx1E67z4UU+TkCXpoN+LtJFY6D/Jyw+qIwpc65heOo4eMIHHiPqzLjHmm04W8Egucq61iIgD2Dd42V3QAeY30c2zicBqmzORbhO1m6bWalbLXUfjF+LlteRmWYZ9ReoytqcNGIT0JdaoqVJN4WmUr1WG+EefTpzi7cgRb14PNsArDoTgTtbOhMiFF72GHOVyTDBI/I/JUKkPRYSFibvjV6hkF4pCV4+RCAuzknoOZSs8SiPgfls= tasks: - kubeadm: null - nvme_loop: null - rook: null - rook.shell: - ceph -s - ceph orch status - ceph orch ps - ceph orch ls - ceph orch device ls - exec: host.a: - "set -ex\ntoolbox() {\n kubectl -n rook-ceph exec -it deploy/rook-ceph-tools\ \ -- \"$@\"\n}\norig_num_osd=`toolbox ceph osd stat | cut -f3 -d \" \"`\ntoolbox\ \ ceph orch osd rm 0 --force\nremoved_pv=\"\"\nwhile [ \"$removed_pv\" = \"\"\ \ ]\ndo\n removed_pv=`kubectl get pv | grep Released | cut -f1 -d \" \"`\n\ \ sleep 3s\ndone\ntarget_path=`kubectl get pv $removed_pv -o jsonpath='{.spec.local.path}'`\n\ host=`echo $removed_pv | cut -f1 -d \"-\"`\ntoolbox ceph orch device zap $host\ \ $target_path --force\nzap_completion=\"0\"\nwhile [ \"$zap_completion\" =\ \ \"0\" ]\ndo\n zap_completion=`kubectl get job -n rook-ceph rook-ceph-device-zap\ \ -o jsonpath='{.status.succeeded.path}'`\n sleep 3s\ndone\nkubectl patch\ \ pv $removed_pv -p '{\"spec\":{\"claimRef\": null}}'\ntoolbox ceph orch apply\ \ osd --all-available-devices\nkubectl delete job rook-ceph-device-zap -n rook-ceph\n\ num_osd=\"0\"\nwhile [ \"$num_osd\" != \"$orig_num_osd\" ]\ndo\n echo \"\ waiting for osd to come back up\"\n num_osd=`toolbox ceph osd stat | cut\ \ -f3 -d \" \"`\n sleep 30s\ndone\n" - rook.shell: commands: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - ceph orch apply rgw foo - ceph orch apply mds foo - ceph orch apply rbd-mirror - ceph orch apply nfs foo --port 12777 teuthology_branch: master teuthology_sha1: d877b023166308f282187444de8e8e725ffb27d2 timestamp: 2022-04-01_18:43:08 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.22301 2022-04-02T02:59:27.879 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a/qa; will attempt to use it 2022-04-02T02:59:27.880 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a/qa/tasks 2022-04-02T02:59:27.880 INFO:teuthology.run_tasks:Running task internal.check_packages... 2022-04-02T02:59:27.882 INFO:teuthology.task.internal:Checking packages... 2022-04-02T02:59:27.903 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a' 2022-04-02T02:59:27.903 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2022-04-02T02:59:27.904 INFO:teuthology.packaging:ref: None 2022-04-02T02:59:27.904 INFO:teuthology.packaging:tag: None 2022-04-02T02:59:27.904 INFO:teuthology.packaging:branch: wip-yuri11-testing-2022-04-01-0842 2022-04-02T02:59:27.905 INFO:teuthology.packaging:sha1: 76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a 2022-04-02T02:59:27.905 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&ref=wip-yuri11-testing-2022-04-01-0842 2022-04-02T02:59:28.070 INFO:teuthology.task.internal:Found packages for ceph version 17.0.0-11449-g76cd7b38-1focal 2022-04-02T02:59:28.071 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2022-04-02T02:59:28.082 INFO:teuthology.task.internal:no buildpackages task found 2022-04-02T02:59:28.082 INFO:teuthology.run_tasks:Running task internal.save_config... 2022-04-02T02:59:28.149 INFO:teuthology.task.internal:Saving configuration 2022-04-02T02:59:28.162 INFO:teuthology.run_tasks:Running task internal.check_lock... 2022-04-02T02:59:28.187 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-04-02T02:59:28.210 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi029.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2022-04-01_18:43:08-rados-wip-yuri11-testing-2022-04-01-0842-distro-default-smithi/6773474', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2022-04-02 02:53:27.704844', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIMbfsKigM6wRALTmnVKYm15n+SlwPJ1y6of0FebmevgJ'} 2022-04-02T02:59:28.230 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi104.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2022-04-01_18:43:08-rados-wip-yuri11-testing-2022-04-01-0842-distro-default-smithi/6773474', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2022-04-02 02:53:27.703676', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIJinMYpEA+6onxR7bCRmjaTI4aakeVhuj1zUqFwRyR8z'} 2022-04-02T02:59:28.251 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi142.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2022-04-01_18:43:08-rados-wip-yuri11-testing-2022-04-01-0842-distro-default-smithi/6773474', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2022-04-02 02:53:27.702335', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIILiZkcQcvxIyQOjgxURdCEA29a9/DKTN9R/T/62ckbF'} 2022-04-02T02:59:28.251 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2022-04-02T02:59:28.261 INFO:teuthology.task.internal:roles: ubuntu@smithi029.front.sepia.ceph.com - ['host.a', 'client.a'] 2022-04-02T02:59:28.262 INFO:teuthology.task.internal:roles: ubuntu@smithi104.front.sepia.ceph.com - ['host.b', 'client.b'] 2022-04-02T02:59:28.262 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['host.c', 'client.c'] 2022-04-02T02:59:28.262 INFO:teuthology.run_tasks:Running task console_log... 2022-04-02T02:59:28.391 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f27137f6c80>, signals=[15]) 2022-04-02T02:59:28.393 INFO:teuthology.run_tasks:Running task internal.connect... 2022-04-02T02:59:28.402 INFO:teuthology.task.internal:Opening connections... 2022-04-02T02:59:28.403 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi029.front.sepia.ceph.com 2022-04-02T02:59:28.404 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi029.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-04-02T02:59:28.458 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi104.front.sepia.ceph.com 2022-04-02T02:59:28.459 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-04-02T02:59:28.511 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi142.front.sepia.ceph.com 2022-04-02T02:59:28.512 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi142.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-04-02T02:59:28.569 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2022-04-02T02:59:28.578 DEBUG:teuthology.orchestra.run.smithi029:> uname -m 2022-04-02T02:59:28.586 INFO:teuthology.orchestra.run.smithi029.stdout:x86_64 2022-04-02T02:59:28.586 DEBUG:teuthology.orchestra.run.smithi029:> cat /etc/os-release 2022-04-02T02:59:28.633 INFO:teuthology.orchestra.run.smithi029.stdout:NAME="Ubuntu" 2022-04-02T02:59:28.634 INFO:teuthology.orchestra.run.smithi029.stdout:VERSION="20.04.3 LTS (Focal Fossa)" 2022-04-02T02:59:28.634 INFO:teuthology.orchestra.run.smithi029.stdout:ID=ubuntu 2022-04-02T02:59:28.634 INFO:teuthology.orchestra.run.smithi029.stdout:ID_LIKE=debian 2022-04-02T02:59:28.634 INFO:teuthology.orchestra.run.smithi029.stdout:PRETTY_NAME="Ubuntu 20.04.3 LTS" 2022-04-02T02:59:28.635 INFO:teuthology.orchestra.run.smithi029.stdout:VERSION_ID="20.04" 2022-04-02T02:59:28.635 INFO:teuthology.orchestra.run.smithi029.stdout:HOME_URL="https://www.ubuntu.com/" 2022-04-02T02:59:28.635 INFO:teuthology.orchestra.run.smithi029.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2022-04-02T02:59:28.635 INFO:teuthology.orchestra.run.smithi029.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2022-04-02T02:59:28.636 INFO:teuthology.orchestra.run.smithi029.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2022-04-02T02:59:28.636 INFO:teuthology.orchestra.run.smithi029.stdout:VERSION_CODENAME=focal 2022-04-02T02:59:28.636 INFO:teuthology.orchestra.run.smithi029.stdout:UBUNTU_CODENAME=focal 2022-04-02T02:59:28.637 INFO:teuthology.lock.ops:Updating smithi029.front.sepia.ceph.com on lock server 2022-04-02T02:59:28.655 DEBUG:teuthology.orchestra.run.smithi104:> uname -m 2022-04-02T02:59:28.662 INFO:teuthology.orchestra.run.smithi104.stdout:x86_64 2022-04-02T02:59:28.663 DEBUG:teuthology.orchestra.run.smithi104:> cat /etc/os-release 2022-04-02T02:59:28.669 INFO:teuthology.orchestra.run.smithi104.stdout:NAME="Ubuntu" 2022-04-02T02:59:28.669 INFO:teuthology.orchestra.run.smithi104.stdout:VERSION="20.04.3 LTS (Focal Fossa)" 2022-04-02T02:59:28.669 INFO:teuthology.orchestra.run.smithi104.stdout:ID=ubuntu 2022-04-02T02:59:28.669 INFO:teuthology.orchestra.run.smithi104.stdout:ID_LIKE=debian 2022-04-02T02:59:28.670 INFO:teuthology.orchestra.run.smithi104.stdout:PRETTY_NAME="Ubuntu 20.04.3 LTS" 2022-04-02T02:59:28.670 INFO:teuthology.orchestra.run.smithi104.stdout:VERSION_ID="20.04" 2022-04-02T02:59:28.670 INFO:teuthology.orchestra.run.smithi104.stdout:HOME_URL="https://www.ubuntu.com/" 2022-04-02T02:59:28.670 INFO:teuthology.orchestra.run.smithi104.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2022-04-02T02:59:28.671 INFO:teuthology.orchestra.run.smithi104.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2022-04-02T02:59:28.671 INFO:teuthology.orchestra.run.smithi104.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2022-04-02T02:59:28.671 INFO:teuthology.orchestra.run.smithi104.stdout:VERSION_CODENAME=focal 2022-04-02T02:59:28.671 INFO:teuthology.orchestra.run.smithi104.stdout:UBUNTU_CODENAME=focal 2022-04-02T02:59:28.672 INFO:teuthology.lock.ops:Updating smithi104.front.sepia.ceph.com on lock server 2022-04-02T02:59:28.688 DEBUG:teuthology.orchestra.run.smithi142:> uname -m 2022-04-02T02:59:28.695 INFO:teuthology.orchestra.run.smithi142.stdout:x86_64 2022-04-02T02:59:28.695 DEBUG:teuthology.orchestra.run.smithi142:> cat /etc/os-release 2022-04-02T02:59:28.744 INFO:teuthology.orchestra.run.smithi142.stdout:NAME="Ubuntu" 2022-04-02T02:59:28.745 INFO:teuthology.orchestra.run.smithi142.stdout:VERSION="20.04.3 LTS (Focal Fossa)" 2022-04-02T02:59:28.745 INFO:teuthology.orchestra.run.smithi142.stdout:ID=ubuntu 2022-04-02T02:59:28.745 INFO:teuthology.orchestra.run.smithi142.stdout:ID_LIKE=debian 2022-04-02T02:59:28.745 INFO:teuthology.orchestra.run.smithi142.stdout:PRETTY_NAME="Ubuntu 20.04.3 LTS" 2022-04-02T02:59:28.746 INFO:teuthology.orchestra.run.smithi142.stdout:VERSION_ID="20.04" 2022-04-02T02:59:28.746 INFO:teuthology.orchestra.run.smithi142.stdout:HOME_URL="https://www.ubuntu.com/" 2022-04-02T02:59:28.746 INFO:teuthology.orchestra.run.smithi142.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2022-04-02T02:59:28.746 INFO:teuthology.orchestra.run.smithi142.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2022-04-02T02:59:28.747 INFO:teuthology.orchestra.run.smithi142.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2022-04-02T02:59:28.747 INFO:teuthology.orchestra.run.smithi142.stdout:VERSION_CODENAME=focal 2022-04-02T02:59:28.747 INFO:teuthology.orchestra.run.smithi142.stdout:UBUNTU_CODENAME=focal 2022-04-02T02:59:28.747 INFO:teuthology.lock.ops:Updating smithi142.front.sepia.ceph.com on lock server 2022-04-02T02:59:28.763 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2022-04-02T02:59:28.776 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2022-04-02T02:59:28.808 INFO:teuthology.task.internal:Checking for old test directory... 2022-04-02T02:59:28.808 DEBUG:teuthology.orchestra.run.smithi029:> test '!' -e /home/ubuntu/cephtest 2022-04-02T02:59:28.811 DEBUG:teuthology.orchestra.run.smithi104:> test '!' -e /home/ubuntu/cephtest 2022-04-02T02:59:28.813 DEBUG:teuthology.orchestra.run.smithi142:> test '!' -e /home/ubuntu/cephtest 2022-04-02T02:59:28.819 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2022-04-02T02:59:28.891 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2022-04-02T02:59:28.891 DEBUG:teuthology.orchestra.run.smithi029:> test -z $(ls -A /var/lib/ceph) 2022-04-02T02:59:28.894 DEBUG:teuthology.orchestra.run.smithi104:> test -z $(ls -A /var/lib/ceph) 2022-04-02T02:59:28.896 DEBUG:teuthology.orchestra.run.smithi142:> test -z $(ls -A /var/lib/ceph) 2022-04-02T02:59:28.915 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2022-04-02T02:59:29.018 INFO:teuthology.run_tasks:Running task kernel... 2022-04-02T02:59:29.051 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2022-04-02T02:59:29.052 INFO:teuthology.task.kernel:normalize config orig: {'hwe': True} 2022-04-02T02:59:29.052 DEBUG:teuthology.task.kernel:normalized overrides {'host.a': {'hwe': True}, 'client.a': {'hwe': True}, 'host.b': {'hwe': True}, 'client.b': {'hwe': True}, 'host.c': {'hwe': True}, 'client.c': {'hwe': True}} 2022-04-02T02:59:29.052 INFO:teuthology.task.kernel:config {'host.a': {'kdb': True, 'sha1': 'distro', 'hwe': True}, 'host.b': {'kdb': True, 'sha1': 'distro', 'hwe': True}, 'host.c': {'kdb': True, 'sha1': 'distro', 'hwe': True}}, timeout 300 2022-04-02T02:59:29.053 DEBUG:teuthology.orchestra.run.smithi029:> test -f /run/.containerenv -o -f /.dockerenv 2022-04-02T02:59:29.054 DEBUG:teuthology.orchestra.run.smithi104:> test -f /run/.containerenv -o -f /.dockerenv 2022-04-02T02:59:29.054 DEBUG:teuthology.orchestra.run.smithi142:> test -f /run/.containerenv -o -f /.dockerenv 2022-04-02T02:59:29.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-04-02T02:59:29.060 DEBUG:teuthology.orchestra.run.smithi104:> uname -r 2022-04-02T02:59:29.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-04-02T02:59:29.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-04-02T02:59:29.063 DEBUG:teuthology.orchestra.run.smithi142:> uname -r 2022-04-02T02:59:29.063 DEBUG:teuthology.orchestra.run.smithi029:> uname -r 2022-04-02T02:59:29.069 INFO:teuthology.orchestra.run.smithi104.stdout:5.4.0-97-generic 2022-04-02T02:59:29.070 INFO:teuthology.task.kernel:Running kernel on smithi104: 5.4.0-97-generic 2022-04-02T02:59:29.070 DEBUG:teuthology.orchestra.run.smithi104:> sudo apt-get clean 2022-04-02T02:59:29.109 INFO:teuthology.orchestra.run.smithi029.stdout:5.4.0-97-generic 2022-04-02T02:59:29.110 INFO:teuthology.task.kernel:Running kernel on smithi029: 5.4.0-97-generic 2022-04-02T02:59:29.110 DEBUG:teuthology.orchestra.run.smithi029:> sudo apt-get clean 2022-04-02T02:59:29.112 INFO:teuthology.orchestra.run.smithi142.stdout:5.4.0-97-generic 2022-04-02T02:59:29.112 INFO:teuthology.task.kernel:Running kernel on smithi142: 5.4.0-97-generic 2022-04-02T02:59:29.113 DEBUG:teuthology.orchestra.run.smithi142:> sudo apt-get clean 2022-04-02T02:59:29.147 DEBUG:teuthology.orchestra.run.smithi104:> sudo apt-get update 2022-04-02T02:59:29.210 DEBUG:teuthology.orchestra.run.smithi029:> sudo apt-get update 2022-04-02T02:59:29.212 DEBUG:teuthology.orchestra.run.smithi142:> sudo apt-get update 2022-04-02T02:59:29.433 INFO:teuthology.orchestra.run.smithi104.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-04-02T02:59:29.434 INFO:teuthology.orchestra.run.smithi104.stdout:Get:2 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2022-04-02T02:59:29.486 INFO:teuthology.orchestra.run.smithi104.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2022-04-02T02:59:29.499 INFO:teuthology.orchestra.run.smithi029.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-04-02T02:59:29.500 INFO:teuthology.orchestra.run.smithi029.stdout:Get:2 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2022-04-02T02:59:29.525 INFO:teuthology.orchestra.run.smithi142.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-04-02T02:59:29.525 INFO:teuthology.orchestra.run.smithi142.stdout:Get:2 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2022-04-02T02:59:29.543 INFO:teuthology.orchestra.run.smithi029.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2022-04-02T02:59:29.573 INFO:teuthology.orchestra.run.smithi142.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2022-04-02T02:59:29.834 INFO:teuthology.orchestra.run.smithi104.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2022-04-02T02:59:29.891 INFO:teuthology.orchestra.run.smithi029.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2022-04-02T02:59:29.905 INFO:teuthology.orchestra.run.smithi104.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [414 kB] 2022-04-02T02:59:29.923 INFO:teuthology.orchestra.run.smithi142.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2022-04-02T02:59:29.969 INFO:teuthology.orchestra.run.smithi029.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [1,377 kB] 2022-04-02T02:59:30.003 INFO:teuthology.orchestra.run.smithi142.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [1,377 kB] 2022-04-02T02:59:30.044 INFO:teuthology.orchestra.run.smithi104.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [626 kB] 2022-04-02T02:59:30.110 INFO:teuthology.orchestra.run.smithi029.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [1,704 kB] 2022-04-02T02:59:30.146 INFO:teuthology.orchestra.run.smithi142.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [1,704 kB] 2022-04-02T02:59:30.211 INFO:teuthology.orchestra.run.smithi104.stdout:Get:7 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [1,377 kB] 2022-04-02T02:59:30.326 INFO:teuthology.orchestra.run.smithi104.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [1,704 kB] 2022-04-02T02:59:30.381 INFO:teuthology.orchestra.run.smithi104.stdout:Get:9 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [238 kB] 2022-04-02T02:59:30.393 INFO:teuthology.orchestra.run.smithi104.stdout:Get:10 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [9,872 B] 2022-04-02T02:59:30.393 INFO:teuthology.orchestra.run.smithi104.stdout:Get:11 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [21.9 kB] 2022-04-02T02:59:30.395 INFO:teuthology.orchestra.run.smithi104.stdout:Get:12 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [853 kB] 2022-04-02T02:59:30.411 INFO:teuthology.orchestra.run.smithi029.stdout:Get:7 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [414 kB] 2022-04-02T02:59:30.420 INFO:teuthology.orchestra.run.smithi142.stdout:Get:7 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [414 kB] 2022-04-02T02:59:30.427 INFO:teuthology.orchestra.run.smithi029.stdout:Get:8 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [238 kB] 2022-04-02T02:59:30.435 INFO:teuthology.orchestra.run.smithi142.stdout:Get:8 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [238 kB] 2022-04-02T02:59:30.436 INFO:teuthology.orchestra.run.smithi029.stdout:Get:9 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [9,872 B] 2022-04-02T02:59:30.436 INFO:teuthology.orchestra.run.smithi029.stdout:Get:10 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [853 kB] 2022-04-02T02:59:30.442 INFO:teuthology.orchestra.run.smithi104.stdout:Get:13 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [122 kB] 2022-04-02T02:59:30.444 INFO:teuthology.orchestra.run.smithi142.stdout:Get:9 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [9,872 B] 2022-04-02T02:59:30.445 INFO:teuthology.orchestra.run.smithi142.stdout:Get:10 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [21.9 kB] 2022-04-02T02:59:30.446 INFO:teuthology.orchestra.run.smithi142.stdout:Get:11 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [853 kB] 2022-04-02T02:59:30.447 INFO:teuthology.orchestra.run.smithi104.stdout:Get:14 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [532 B] 2022-04-02T02:59:30.447 INFO:teuthology.orchestra.run.smithi104.stdout:Get:15 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [695 kB] 2022-04-02T02:59:30.474 INFO:teuthology.orchestra.run.smithi104.stdout:Get:16 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [547 kB] 2022-04-02T02:59:30.479 INFO:teuthology.orchestra.run.smithi104.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [319 kB] 2022-04-02T02:59:30.489 INFO:teuthology.orchestra.run.smithi104.stdout:Get:18 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [122 kB] 2022-04-02T02:59:30.492 INFO:teuthology.orchestra.run.smithi104.stdout:Get:19 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [14.1 kB] 2022-04-02T02:59:30.493 INFO:teuthology.orchestra.run.smithi029.stdout:Get:11 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [21.9 kB] 2022-04-02T02:59:30.494 INFO:teuthology.orchestra.run.smithi029.stdout:Get:12 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [122 kB] 2022-04-02T02:59:30.495 INFO:teuthology.orchestra.run.smithi104.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [14.8 kB] 2022-04-02T02:59:30.495 INFO:teuthology.orchestra.run.smithi104.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [24.3 kB] 2022-04-02T02:59:30.496 INFO:teuthology.orchestra.run.smithi104.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [913 kB] 2022-04-02T02:59:30.498 INFO:teuthology.orchestra.run.smithi029.stdout:Get:13 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [532 B] 2022-04-02T02:59:30.499 INFO:teuthology.orchestra.run.smithi029.stdout:Get:14 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [547 kB] 2022-04-02T02:59:30.502 INFO:teuthology.orchestra.run.smithi142.stdout:Get:12 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [122 kB] 2022-04-02T02:59:30.502 INFO:teuthology.orchestra.run.smithi029.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [626 kB] 2022-04-02T02:59:30.507 INFO:teuthology.orchestra.run.smithi142.stdout:Get:13 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [532 B] 2022-04-02T02:59:30.507 INFO:teuthology.orchestra.run.smithi142.stdout:Get:14 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [695 kB] 2022-04-02T02:59:30.519 INFO:teuthology.orchestra.run.smithi029.stdout:Get:16 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [695 kB] 2022-04-02T02:59:30.521 INFO:teuthology.orchestra.run.smithi029.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [319 kB] 2022-04-02T02:59:30.532 INFO:teuthology.orchestra.run.smithi029.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [14.8 kB] 2022-04-02T02:59:30.532 INFO:teuthology.orchestra.run.smithi104.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [130 kB] 2022-04-02T02:59:30.533 INFO:teuthology.orchestra.run.smithi029.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [24.3 kB] 2022-04-02T02:59:30.533 INFO:teuthology.orchestra.run.smithi029.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [913 kB] 2022-04-02T02:59:30.534 INFO:teuthology.orchestra.run.smithi142.stdout:Get:15 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [547 kB] 2022-04-02T02:59:30.537 INFO:teuthology.orchestra.run.smithi104.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [528 B] 2022-04-02T02:59:30.537 INFO:teuthology.orchestra.run.smithi104.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [915 kB] 2022-04-02T02:59:30.539 INFO:teuthology.orchestra.run.smithi142.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [626 kB] 2022-04-02T02:59:30.542 INFO:teuthology.orchestra.run.smithi104.stdout:Get:26 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [20.7 kB] 2022-04-02T02:59:30.543 INFO:teuthology.orchestra.run.smithi104.stdout:Get:27 http://security.ubuntu.com/ubuntu focal-security/multiverse i386 Packages [7,180 B] 2022-04-02T02:59:30.543 INFO:teuthology.orchestra.run.smithi104.stdout:Get:28 http://security.ubuntu.com/ubuntu focal-security/multiverse Translation-en [5,196 B] 2022-04-02T02:59:30.544 INFO:teuthology.orchestra.run.smithi104.stdout:Get:29 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 c-n-f Metadata [500 B] 2022-04-02T02:59:30.545 INFO:teuthology.orchestra.run.smithi029.stdout:Get:21 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [122 kB] 2022-04-02T02:59:30.549 INFO:teuthology.orchestra.run.smithi029.stdout:Get:22 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [14.1 kB] 2022-04-02T02:59:30.554 INFO:teuthology.orchestra.run.smithi142.stdout:Get:17 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [122 kB] 2022-04-02T02:59:30.557 INFO:teuthology.orchestra.run.smithi142.stdout:Get:18 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [14.1 kB] 2022-04-02T02:59:30.558 INFO:teuthology.orchestra.run.smithi142.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [319 kB] 2022-04-02T02:59:30.569 INFO:teuthology.orchestra.run.smithi142.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [14.8 kB] 2022-04-02T02:59:30.569 INFO:teuthology.orchestra.run.smithi104.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [676 kB] 2022-04-02T02:59:30.570 INFO:teuthology.orchestra.run.smithi142.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [913 kB] 2022-04-02T02:59:30.585 INFO:teuthology.orchestra.run.smithi104.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [204 kB] 2022-04-02T02:59:30.586 INFO:teuthology.orchestra.run.smithi029.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [130 kB] 2022-04-02T02:59:30.589 INFO:teuthology.orchestra.run.smithi029.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [528 B] 2022-04-02T02:59:30.590 INFO:teuthology.orchestra.run.smithi104.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [20.4 kB] 2022-04-02T02:59:30.591 INFO:teuthology.orchestra.run.smithi029.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [676 kB] 2022-04-02T02:59:30.592 INFO:teuthology.orchestra.run.smithi029.stdout:Get:26 http://security.ubuntu.com/ubuntu focal-security/multiverse i386 Packages [7,180 B] 2022-04-02T02:59:30.593 INFO:teuthology.orchestra.run.smithi029.stdout:Get:27 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [20.7 kB] 2022-04-02T02:59:30.594 INFO:teuthology.orchestra.run.smithi029.stdout:Get:28 http://security.ubuntu.com/ubuntu focal-security/multiverse Translation-en [5,196 B] 2022-04-02T02:59:30.594 INFO:teuthology.orchestra.run.smithi029.stdout:Get:29 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 c-n-f Metadata [500 B] 2022-04-02T02:59:30.598 INFO:teuthology.orchestra.run.smithi142.stdout:Get:22 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [20.7 kB] 2022-04-02T02:59:30.598 INFO:teuthology.orchestra.run.smithi142.stdout:Get:23 http://security.ubuntu.com/ubuntu focal-security/multiverse i386 Packages [7,180 B] 2022-04-02T02:59:30.599 INFO:teuthology.orchestra.run.smithi142.stdout:Get:24 http://security.ubuntu.com/ubuntu focal-security/multiverse Translation-en [5,196 B] 2022-04-02T02:59:30.599 INFO:teuthology.orchestra.run.smithi142.stdout:Get:25 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 c-n-f Metadata [500 B] 2022-04-02T02:59:30.610 INFO:teuthology.orchestra.run.smithi029.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [915 kB] 2022-04-02T02:59:30.617 INFO:teuthology.orchestra.run.smithi104.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8,432 B] 2022-04-02T02:59:30.618 INFO:teuthology.orchestra.run.smithi104.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [24.4 kB] 2022-04-02T02:59:30.618 INFO:teuthology.orchestra.run.smithi104.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [7,336 B] 2022-04-02T02:59:30.619 INFO:teuthology.orchestra.run.smithi104.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [592 B] 2022-04-02T02:59:30.619 INFO:teuthology.orchestra.run.smithi104.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [42.2 kB] 2022-04-02T02:59:30.622 INFO:teuthology.orchestra.run.smithi142.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [24.3 kB] 2022-04-02T02:59:30.623 INFO:teuthology.orchestra.run.smithi142.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [130 kB] 2022-04-02T02:59:30.626 INFO:teuthology.orchestra.run.smithi142.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [528 B] 2022-04-02T02:59:30.627 INFO:teuthology.orchestra.run.smithi142.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [915 kB] 2022-04-02T02:59:30.642 INFO:teuthology.orchestra.run.smithi104.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal-backports/main i386 Packages [34.8 kB] 2022-04-02T02:59:30.655 INFO:teuthology.orchestra.run.smithi142.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [676 kB] 2022-04-02T02:59:30.675 INFO:teuthology.orchestra.run.smithi142.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [204 kB] 2022-04-02T02:59:30.680 INFO:teuthology.orchestra.run.smithi142.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [20.4 kB] 2022-04-02T02:59:30.707 INFO:teuthology.orchestra.run.smithi029.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [204 kB] 2022-04-02T02:59:30.708 INFO:teuthology.orchestra.run.smithi142.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8,432 B] 2022-04-02T02:59:30.709 INFO:teuthology.orchestra.run.smithi142.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [24.4 kB] 2022-04-02T02:59:30.709 INFO:teuthology.orchestra.run.smithi142.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [7,336 B] 2022-04-02T02:59:30.710 INFO:teuthology.orchestra.run.smithi142.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [592 B] 2022-04-02T02:59:30.710 INFO:teuthology.orchestra.run.smithi142.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-backports/main i386 Packages [34.8 kB] 2022-04-02T02:59:30.711 INFO:teuthology.orchestra.run.smithi029.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [20.4 kB] 2022-04-02T02:59:30.711 INFO:teuthology.orchestra.run.smithi029.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [24.4 kB] 2022-04-02T02:59:30.712 INFO:teuthology.orchestra.run.smithi029.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8,432 B] 2022-04-02T02:59:30.712 INFO:teuthology.orchestra.run.smithi029.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [7,336 B] 2022-04-02T02:59:30.712 INFO:teuthology.orchestra.run.smithi029.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [592 B] 2022-04-02T02:59:30.712 INFO:teuthology.orchestra.run.smithi029.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [42.2 kB] 2022-04-02T02:59:30.712 INFO:teuthology.orchestra.run.smithi029.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal-backports/main i386 Packages [34.8 kB] 2022-04-02T02:59:30.721 INFO:teuthology.orchestra.run.smithi142.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [42.2 kB] 2022-04-02T02:59:30.722 INFO:teuthology.orchestra.run.smithi104.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal-backports/main Translation-en [10.1 kB] 2022-04-02T02:59:30.722 INFO:teuthology.orchestra.run.smithi104.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [22.7 kB] 2022-04-02T02:59:30.723 INFO:teuthology.orchestra.run.smithi104.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [12.9 kB] 2022-04-02T02:59:30.723 INFO:teuthology.orchestra.run.smithi104.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal-backports/universe Translation-en [15.5 kB] 2022-04-02T02:59:30.724 INFO:teuthology.orchestra.run.smithi104.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [808 B] 2022-04-02T02:59:30.764 INFO:teuthology.orchestra.run.smithi029.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal-backports/main Translation-en [10.1 kB] 2022-04-02T02:59:30.801 INFO:teuthology.orchestra.run.smithi142.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal-backports/main Translation-en [10.1 kB] 2022-04-02T02:59:30.801 INFO:teuthology.orchestra.run.smithi142.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [22.7 kB] 2022-04-02T02:59:30.802 INFO:teuthology.orchestra.run.smithi142.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [12.9 kB] 2022-04-02T02:59:30.802 INFO:teuthology.orchestra.run.smithi142.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal-backports/universe Translation-en [15.5 kB] 2022-04-02T02:59:30.803 INFO:teuthology.orchestra.run.smithi142.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [808 B] 2022-04-02T02:59:30.843 INFO:teuthology.orchestra.run.smithi029.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [12.9 kB] 2022-04-02T02:59:30.844 INFO:teuthology.orchestra.run.smithi029.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [22.7 kB] 2022-04-02T02:59:30.845 INFO:teuthology.orchestra.run.smithi029.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal-backports/universe Translation-en [15.5 kB] 2022-04-02T02:59:30.845 INFO:teuthology.orchestra.run.smithi029.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [808 B] 2022-04-02T02:59:33.422 INFO:teuthology.orchestra.run.smithi104.stdout:Fetched 10.5 MB in 2s (4,754 kB/s) 2022-04-02T02:59:33.604 INFO:teuthology.orchestra.run.smithi142.stdout:Fetched 10.5 MB in 2s (4,783 kB/s) 2022-04-02T02:59:33.733 INFO:teuthology.orchestra.run.smithi029.stdout:Fetched 10.5 MB in 2s (4,661 kB/s) 2022-04-02T02:59:34.569 INFO:teuthology.orchestra.run.smithi104.stdout:Reading package lists... 2022-04-02T02:59:34.593 DEBUG:teuthology.orchestra.run.smithi104:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic-hwe-20.04 2022-04-02T02:59:34.661 INFO:teuthology.orchestra.run.smithi104.stdout:Reading package lists... 2022-04-02T02:59:34.814 INFO:teuthology.orchestra.run.smithi142.stdout:Reading package lists... 2022-04-02T02:59:34.836 INFO:teuthology.orchestra.run.smithi104.stdout:Building dependency tree... 2022-04-02T02:59:34.837 INFO:teuthology.orchestra.run.smithi104.stdout:Reading state information... 2022-04-02T02:59:34.838 DEBUG:teuthology.orchestra.run.smithi142:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic-hwe-20.04 2022-04-02T02:59:34.903 INFO:teuthology.orchestra.run.smithi142.stdout:Reading package lists... 2022-04-02T02:59:34.987 INFO:teuthology.orchestra.run.smithi104.stdout:The following additional packages will be installed: 2022-04-02T02:59:34.987 INFO:teuthology.orchestra.run.smithi104.stdout: libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 2022-04-02T02:59:34.988 INFO:teuthology.orchestra.run.smithi104.stdout: libusbmuxd6 linux-image-5.13.0-39-generic linux-modules-5.13.0-39-generic 2022-04-02T02:59:34.988 INFO:teuthology.orchestra.run.smithi104.stdout: linux-modules-extra-5.13.0-39-generic thermald upower usbmuxd 2022-04-02T02:59:34.989 INFO:teuthology.orchestra.run.smithi104.stdout:Suggested packages: 2022-04-02T02:59:34.989 INFO:teuthology.orchestra.run.smithi104.stdout: libusbmuxd-tools fdutils linux-doc | linux-hwe-5.13-source-5.13.0 2022-04-02T02:59:34.989 INFO:teuthology.orchestra.run.smithi104.stdout: linux-hwe-5.13-tools linux-headers-5.13.0-39-generic 2022-04-02T02:59:35.040 INFO:teuthology.orchestra.run.smithi104.stdout:The following NEW packages will be installed: 2022-04-02T02:59:35.041 INFO:teuthology.orchestra.run.smithi104.stdout: libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 2022-04-02T02:59:35.041 INFO:teuthology.orchestra.run.smithi104.stdout: libusbmuxd6 linux-image-5.13.0-39-generic linux-image-generic-hwe-20.04 2022-04-02T02:59:35.041 INFO:teuthology.orchestra.run.smithi104.stdout: linux-modules-5.13.0-39-generic linux-modules-extra-5.13.0-39-generic 2022-04-02T02:59:35.041 INFO:teuthology.orchestra.run.smithi104.stdout: thermald upower usbmuxd 2022-04-02T02:59:35.063 INFO:teuthology.orchestra.run.smithi142.stdout:Building dependency tree... 2022-04-02T02:59:35.064 INFO:teuthology.orchestra.run.smithi142.stdout:Reading state information... 2022-04-02T02:59:35.106 INFO:teuthology.orchestra.run.smithi104.stdout:0 upgraded, 13 newly installed, 0 to remove and 120 not upgraded. 2022-04-02T02:59:35.106 INFO:teuthology.orchestra.run.smithi104.stdout:Need to get 79.9 MB of archives. 2022-04-02T02:59:35.106 INFO:teuthology.orchestra.run.smithi104.stdout:After this operation, 410 MB of additional disk space will be used. 2022-04-02T02:59:35.107 INFO:teuthology.orchestra.run.smithi104.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 libdbus-glib-1-2 amd64 0.110-5fakssync1 [59.1 kB] 2022-04-02T02:59:35.177 INFO:teuthology.orchestra.run.smithi104.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 libplist3 amd64 2.1.0-4build2 [31.6 kB] 2022-04-02T02:59:35.178 INFO:teuthology.orchestra.run.smithi104.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 libusbmuxd6 amd64 2.0.1-2 [19.1 kB] 2022-04-02T02:59:35.179 INFO:teuthology.orchestra.run.smithi104.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 libimobiledevice6 amd64 1.2.1~git20191129.9f79242-1build1 [65.2 kB] 2022-04-02T02:59:35.197 INFO:teuthology.orchestra.run.smithi142.stdout:The following additional packages will be installed: 2022-04-02T02:59:35.197 INFO:teuthology.orchestra.run.smithi142.stdout: libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 2022-04-02T02:59:35.197 INFO:teuthology.orchestra.run.smithi142.stdout: libusbmuxd6 linux-image-5.13.0-39-generic linux-modules-5.13.0-39-generic 2022-04-02T02:59:35.198 INFO:teuthology.orchestra.run.smithi142.stdout: linux-modules-extra-5.13.0-39-generic thermald upower usbmuxd 2022-04-02T02:59:35.198 INFO:teuthology.orchestra.run.smithi142.stdout:Suggested packages: 2022-04-02T02:59:35.198 INFO:teuthology.orchestra.run.smithi142.stdout: libusbmuxd-tools fdutils linux-doc | linux-hwe-5.13-source-5.13.0 2022-04-02T02:59:35.199 INFO:teuthology.orchestra.run.smithi142.stdout: linux-hwe-5.13-tools linux-headers-5.13.0-39-generic 2022-04-02T02:59:35.203 INFO:teuthology.orchestra.run.smithi104.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 libupower-glib3 amd64 0.99.11-1build2 [43.2 kB] 2022-04-02T02:59:35.208 INFO:teuthology.orchestra.run.smithi104.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.13.0-39-generic amd64 5.13.0-39.44~20.04.1 [18.3 MB] 2022-04-02T02:59:35.250 INFO:teuthology.orchestra.run.smithi142.stdout:The following NEW packages will be installed: 2022-04-02T02:59:35.250 INFO:teuthology.orchestra.run.smithi142.stdout: libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 2022-04-02T02:59:35.250 INFO:teuthology.orchestra.run.smithi142.stdout: libusbmuxd6 linux-image-5.13.0-39-generic linux-image-generic-hwe-20.04 2022-04-02T02:59:35.251 INFO:teuthology.orchestra.run.smithi142.stdout: linux-modules-5.13.0-39-generic linux-modules-extra-5.13.0-39-generic 2022-04-02T02:59:35.251 INFO:teuthology.orchestra.run.smithi142.stdout: thermald upower usbmuxd 2022-04-02T02:59:35.345 INFO:teuthology.orchestra.run.smithi142.stdout:0 upgraded, 13 newly installed, 0 to remove and 120 not upgraded. 2022-04-02T02:59:35.345 INFO:teuthology.orchestra.run.smithi142.stdout:Need to get 79.9 MB of archives. 2022-04-02T02:59:35.345 INFO:teuthology.orchestra.run.smithi142.stdout:After this operation, 410 MB of additional disk space will be used. 2022-04-02T02:59:35.346 INFO:teuthology.orchestra.run.smithi142.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 libdbus-glib-1-2 amd64 0.110-5fakssync1 [59.1 kB] 2022-04-02T02:59:35.437 INFO:teuthology.orchestra.run.smithi142.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 libplist3 amd64 2.1.0-4build2 [31.6 kB] 2022-04-02T02:59:35.452 INFO:teuthology.orchestra.run.smithi142.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 libusbmuxd6 amd64 2.0.1-2 [19.1 kB] 2022-04-02T02:59:35.458 INFO:teuthology.orchestra.run.smithi142.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 libimobiledevice6 amd64 1.2.1~git20191129.9f79242-1build1 [65.2 kB] 2022-04-02T02:59:35.480 INFO:teuthology.orchestra.run.smithi142.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 libupower-glib3 amd64 0.99.11-1build2 [43.2 kB] 2022-04-02T02:59:35.490 INFO:teuthology.orchestra.run.smithi142.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.13.0-39-generic amd64 5.13.0-39.44~20.04.1 [18.3 MB] 2022-04-02T02:59:35.528 INFO:teuthology.orchestra.run.smithi104.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.13.0-39-generic amd64 5.13.0-39.44~20.04.1 [10.0 MB] 2022-04-02T02:59:35.577 INFO:teuthology.orchestra.run.smithi029.stdout:Reading package lists... 2022-04-02T02:59:35.601 DEBUG:teuthology.orchestra.run.smithi029:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic-hwe-20.04 2022-04-02T02:59:35.656 INFO:teuthology.orchestra.run.smithi104.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.13.0-39-generic amd64 5.13.0-39.44~20.04.1 [51.0 MB] 2022-04-02T02:59:35.668 INFO:teuthology.orchestra.run.smithi029.stdout:Reading package lists... 2022-04-02T02:59:35.848 INFO:teuthology.orchestra.run.smithi029.stdout:Building dependency tree... 2022-04-02T02:59:35.849 INFO:teuthology.orchestra.run.smithi029.stdout:Reading state information... 2022-04-02T02:59:35.951 INFO:teuthology.orchestra.run.smithi142.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.13.0-39-generic amd64 5.13.0-39.44~20.04.1 [10.0 MB] 2022-04-02T02:59:36.001 INFO:teuthology.orchestra.run.smithi029.stdout:The following additional packages will be installed: 2022-04-02T02:59:36.001 INFO:teuthology.orchestra.run.smithi029.stdout: libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 2022-04-02T02:59:36.002 INFO:teuthology.orchestra.run.smithi029.stdout: libusbmuxd6 linux-image-5.13.0-39-generic linux-modules-5.13.0-39-generic 2022-04-02T02:59:36.002 INFO:teuthology.orchestra.run.smithi029.stdout: linux-modules-extra-5.13.0-39-generic thermald upower usbmuxd 2022-04-02T02:59:36.003 INFO:teuthology.orchestra.run.smithi029.stdout:Suggested packages: 2022-04-02T02:59:36.003 INFO:teuthology.orchestra.run.smithi029.stdout: libusbmuxd-tools fdutils linux-doc | linux-hwe-5.13-source-5.13.0 2022-04-02T02:59:36.003 INFO:teuthology.orchestra.run.smithi029.stdout: linux-hwe-5.13-tools linux-headers-5.13.0-39-generic 2022-04-02T02:59:36.055 INFO:teuthology.orchestra.run.smithi029.stdout:The following NEW packages will be installed: 2022-04-02T02:59:36.056 INFO:teuthology.orchestra.run.smithi029.stdout: libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 2022-04-02T02:59:36.056 INFO:teuthology.orchestra.run.smithi029.stdout: libusbmuxd6 linux-image-5.13.0-39-generic linux-image-generic-hwe-20.04 2022-04-02T02:59:36.056 INFO:teuthology.orchestra.run.smithi029.stdout: linux-modules-5.13.0-39-generic linux-modules-extra-5.13.0-39-generic 2022-04-02T02:59:36.056 INFO:teuthology.orchestra.run.smithi029.stdout: thermald upower usbmuxd 2022-04-02T02:59:36.111 INFO:teuthology.orchestra.run.smithi142.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.13.0-39-generic amd64 5.13.0-39.44~20.04.1 [51.0 MB] 2022-04-02T02:59:36.155 INFO:teuthology.orchestra.run.smithi029.stdout:0 upgraded, 13 newly installed, 0 to remove and 120 not upgraded. 2022-04-02T02:59:36.155 INFO:teuthology.orchestra.run.smithi029.stdout:Need to get 79.9 MB of archives. 2022-04-02T02:59:36.155 INFO:teuthology.orchestra.run.smithi029.stdout:After this operation, 410 MB of additional disk space will be used. 2022-04-02T02:59:36.155 INFO:teuthology.orchestra.run.smithi029.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 libdbus-glib-1-2 amd64 0.110-5fakssync1 [59.1 kB] 2022-04-02T02:59:36.337 INFO:teuthology.orchestra.run.smithi029.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 libplist3 amd64 2.1.0-4build2 [31.6 kB] 2022-04-02T02:59:36.354 INFO:teuthology.orchestra.run.smithi029.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 libusbmuxd6 amd64 2.0.1-2 [19.1 kB] 2022-04-02T02:59:36.355 INFO:teuthology.orchestra.run.smithi029.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 libimobiledevice6 amd64 1.2.1~git20191129.9f79242-1build1 [65.2 kB] 2022-04-02T02:59:36.403 INFO:teuthology.orchestra.run.smithi029.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 libupower-glib3 amd64 0.99.11-1build2 [43.2 kB] 2022-04-02T02:59:36.412 INFO:teuthology.orchestra.run.smithi029.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.13.0-39-generic amd64 5.13.0-39.44~20.04.1 [18.3 MB] 2022-04-02T02:59:36.729 INFO:teuthology.orchestra.run.smithi104.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic-hwe-20.04 amd64 5.13.0.39.44~20.04.24 [2,620 B] 2022-04-02T02:59:36.729 INFO:teuthology.orchestra.run.smithi104.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libevdev2 amd64 1.9.0+dfsg-1ubuntu0.1 [31.6 kB] 2022-04-02T02:59:36.730 INFO:teuthology.orchestra.run.smithi104.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 thermald amd64 1.9.1-1ubuntu0.6 [233 kB] 2022-04-02T02:59:36.736 INFO:teuthology.orchestra.run.smithi104.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/main amd64 upower amd64 0.99.11-1build2 [104 kB] 2022-04-02T02:59:36.738 INFO:teuthology.orchestra.run.smithi104.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 usbmuxd amd64 1.1.1~git20191130.9af2b12-1 [38.4 kB] 2022-04-02T02:59:37.002 INFO:teuthology.orchestra.run.smithi029.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.13.0-39-generic amd64 5.13.0-39.44~20.04.1 [10.0 MB] 2022-04-02T02:59:37.101 INFO:teuthology.orchestra.run.smithi029.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.13.0-39-generic amd64 5.13.0-39.44~20.04.1 [51.0 MB] 2022-04-02T02:59:37.144 INFO:teuthology.orchestra.run.smithi104.stdout:Fetched 79.9 MB in 2s (47.3 MB/s) 2022-04-02T02:59:37.250 INFO:teuthology.orchestra.run.smithi142.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic-hwe-20.04 amd64 5.13.0.39.44~20.04.24 [2,620 B] 2022-04-02T02:59:37.250 INFO:teuthology.orchestra.run.smithi142.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libevdev2 amd64 1.9.0+dfsg-1ubuntu0.1 [31.6 kB] 2022-04-02T02:59:37.251 INFO:teuthology.orchestra.run.smithi142.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 thermald amd64 1.9.1-1ubuntu0.6 [233 kB] 2022-04-02T02:59:37.255 INFO:teuthology.orchestra.run.smithi142.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/main amd64 upower amd64 0.99.11-1build2 [104 kB] 2022-04-02T02:59:37.256 INFO:teuthology.orchestra.run.smithi142.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 usbmuxd amd64 1.1.1~git20191130.9af2b12-1 [38.4 kB] 2022-04-02T02:59:37.341 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package libdbus-glib-1-2:amd64. 2022-04-02T02:59:37.732 INFO:teuthology.orchestra.run.smithi142.stdout:Fetched 79.9 MB in 2s (39.0 MB/s) 2022-04-02T02:59:37.948 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package libdbus-glib-1-2:amd64. 2022-04-02T02:59:38.028 INFO:teuthology.orchestra.run.smithi029.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic-hwe-20.04 amd64 5.13.0.39.44~20.04.24 [2,620 B] 2022-04-02T02:59:38.029 INFO:teuthology.orchestra.run.smithi029.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libevdev2 amd64 1.9.0+dfsg-1ubuntu0.1 [31.6 kB] 2022-04-02T02:59:38.030 INFO:teuthology.orchestra.run.smithi029.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 thermald amd64 1.9.1-1ubuntu0.6 [233 kB] 2022-04-02T02:59:38.036 INFO:teuthology.orchestra.run.smithi029.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/main amd64 upower amd64 0.99.11-1build2 [104 kB] 2022-04-02T02:59:38.038 INFO:teuthology.orchestra.run.smithi029.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 usbmuxd amd64 1.1.1~git20191130.9af2b12-1 [38.4 kB] 2022-04-02T02:59:38.557 INFO:teuthology.orchestra.run.smithi029.stdout:Fetched 79.9 MB in 2s (40.5 MB/s) 2022-04-02T02:59:38.772 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package libdbus-glib-1-2:amd64. 2022-04-02T02:59:38.808 INFO:teuthology.orchestra.run.smithi104.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 102042 files and directories currently installed.) 2022-04-02T02:59:38.813 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../00-libdbus-glib-1-2_0.110-5fakssync1_amd64.deb ... 2022-04-02T02:59:38.929 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking libdbus-glib-1-2:amd64 (0.110-5fakssync1) ... 2022-04-02T02:59:39.299 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package libplist3:amd64. 2022-04-02T02:59:39.314 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../01-libplist3_2.1.0-4build2_amd64.deb ... 2022-04-02T02:59:39.357 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking libplist3:amd64 (2.1.0-4build2) ... 2022-04-02T02:59:39.419 INFO:teuthology.orchestra.run.smithi142.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 102042 files and directories currently installed.) 2022-04-02T02:59:39.423 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../00-libdbus-glib-1-2_0.110-5fakssync1_amd64.deb ... 2022-04-02T02:59:39.603 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking libdbus-glib-1-2:amd64 (0.110-5fakssync1) ... 2022-04-02T02:59:39.685 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package libusbmuxd6:amd64. 2022-04-02T02:59:39.699 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../02-libusbmuxd6_2.0.1-2_amd64.deb ... 2022-04-02T02:59:39.760 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking libusbmuxd6:amd64 (2.0.1-2) ... 2022-04-02T02:59:39.980 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package libplist3:amd64. 2022-04-02T02:59:39.994 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../01-libplist3_2.1.0-4build2_amd64.deb ... 2022-04-02T02:59:40.038 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package libimobiledevice6:amd64. 2022-04-02T02:59:40.038 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking libplist3:amd64 (2.1.0-4build2) ... 2022-04-02T02:59:40.053 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../03-libimobiledevice6_1.2.1~git20191129.9f79242-1build1_amd64.deb ... 2022-04-02T02:59:40.096 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking libimobiledevice6:amd64 (1.2.1~git20191129.9f79242-1build1) ... 2022-04-02T02:59:40.391 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package libusbmuxd6:amd64. 2022-04-02T02:59:40.404 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../02-libusbmuxd6_2.0.1-2_amd64.deb ... 2022-04-02T02:59:40.441 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking libusbmuxd6:amd64 (2.0.1-2) ... 2022-04-02T02:59:40.441 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package libupower-glib3:amd64. 2022-04-02T02:59:40.455 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../04-libupower-glib3_0.99.11-1build2_amd64.deb ... 2022-04-02T02:59:40.499 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking libupower-glib3:amd64 (0.99.11-1build2) ... 2022-04-02T02:59:40.802 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package linux-modules-5.13.0-39-generic. 2022-04-02T02:59:40.817 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../05-linux-modules-5.13.0-39-generic_5.13.0-39.44~20.04.1_amd64.deb ... 2022-04-02T02:59:40.821 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package libimobiledevice6:amd64. 2022-04-02T02:59:40.835 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../03-libimobiledevice6_1.2.1~git20191129.9f79242-1build1_amd64.deb ... 2022-04-02T02:59:40.860 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking linux-modules-5.13.0-39-generic (5.13.0-39.44~20.04.1) ... 2022-04-02T02:59:40.879 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking libimobiledevice6:amd64 (1.2.1~git20191129.9f79242-1build1) ... 2022-04-02T02:59:41.036 INFO:teuthology.orchestra.run.smithi029.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 102042 files and directories currently installed.) 2022-04-02T02:59:41.041 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../00-libdbus-glib-1-2_0.110-5fakssync1_amd64.deb ... 2022-04-02T02:59:41.154 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking libdbus-glib-1-2:amd64 (0.110-5fakssync1) ... 2022-04-02T02:59:41.240 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package libupower-glib3:amd64. 2022-04-02T02:59:41.252 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../04-libupower-glib3_0.99.11-1build2_amd64.deb ... 2022-04-02T02:59:41.290 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking libupower-glib3:amd64 (0.99.11-1build2) ... 2022-04-02T02:59:41.534 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package libplist3:amd64. 2022-04-02T02:59:41.547 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../01-libplist3_2.1.0-4build2_amd64.deb ... 2022-04-02T02:59:41.592 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking libplist3:amd64 (2.1.0-4build2) ... 2022-04-02T02:59:41.626 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package linux-modules-5.13.0-39-generic. 2022-04-02T02:59:41.639 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../05-linux-modules-5.13.0-39-generic_5.13.0-39.44~20.04.1_amd64.deb ... 2022-04-02T02:59:41.693 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking linux-modules-5.13.0-39-generic (5.13.0-39.44~20.04.1) ... 2022-04-02T02:59:41.932 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package libusbmuxd6:amd64. 2022-04-02T02:59:41.948 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../02-libusbmuxd6_2.0.1-2_amd64.deb ... 2022-04-02T02:59:41.991 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking libusbmuxd6:amd64 (2.0.1-2) ... 2022-04-02T02:59:42.373 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package libimobiledevice6:amd64. 2022-04-02T02:59:42.387 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../03-libimobiledevice6_1.2.1~git20191129.9f79242-1build1_amd64.deb ... 2022-04-02T02:59:42.440 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking libimobiledevice6:amd64 (1.2.1~git20191129.9f79242-1build1) ... 2022-04-02T02:59:42.828 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package libupower-glib3:amd64. 2022-04-02T02:59:42.844 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../04-libupower-glib3_0.99.11-1build2_amd64.deb ... 2022-04-02T02:59:42.887 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking libupower-glib3:amd64 (0.99.11-1build2) ... 2022-04-02T02:59:43.085 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package linux-image-5.13.0-39-generic. 2022-04-02T02:59:43.098 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../06-linux-image-5.13.0-39-generic_5.13.0-39.44~20.04.1_amd64.deb ... 2022-04-02T02:59:43.176 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking linux-image-5.13.0-39-generic (5.13.0-39.44~20.04.1) ... 2022-04-02T02:59:43.219 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package linux-modules-5.13.0-39-generic. 2022-04-02T02:59:43.232 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../05-linux-modules-5.13.0-39-generic_5.13.0-39.44~20.04.1_amd64.deb ... 2022-04-02T02:59:43.277 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking linux-modules-5.13.0-39-generic (5.13.0-39.44~20.04.1) ... 2022-04-02T02:59:43.638 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package linux-modules-extra-5.13.0-39-generic. 2022-04-02T02:59:43.651 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../07-linux-modules-extra-5.13.0-39-generic_5.13.0-39.44~20.04.1_amd64.deb ... 2022-04-02T02:59:43.679 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking linux-modules-extra-5.13.0-39-generic (5.13.0-39.44~20.04.1) ... 2022-04-02T02:59:43.935 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package linux-image-5.13.0-39-generic. 2022-04-02T02:59:43.948 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../06-linux-image-5.13.0-39-generic_5.13.0-39.44~20.04.1_amd64.deb ... 2022-04-02T02:59:44.027 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking linux-image-5.13.0-39-generic (5.13.0-39.44~20.04.1) ... 2022-04-02T02:59:44.530 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package linux-modules-extra-5.13.0-39-generic. 2022-04-02T02:59:44.543 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../07-linux-modules-extra-5.13.0-39-generic_5.13.0-39.44~20.04.1_amd64.deb ... 2022-04-02T02:59:44.613 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking linux-modules-extra-5.13.0-39-generic (5.13.0-39.44~20.04.1) ... 2022-04-02T02:59:45.690 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package linux-image-5.13.0-39-generic. 2022-04-02T02:59:45.706 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../06-linux-image-5.13.0-39-generic_5.13.0-39.44~20.04.1_amd64.deb ... 2022-04-02T02:59:45.849 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking linux-image-5.13.0-39-generic (5.13.0-39.44~20.04.1) ... 2022-04-02T02:59:46.406 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package linux-modules-extra-5.13.0-39-generic. 2022-04-02T02:59:46.422 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../07-linux-modules-extra-5.13.0-39-generic_5.13.0-39.44~20.04.1_amd64.deb ... 2022-04-02T02:59:46.464 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking linux-modules-extra-5.13.0-39-generic (5.13.0-39.44~20.04.1) ... 2022-04-02T02:59:49.437 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package linux-image-generic-hwe-20.04. 2022-04-02T02:59:49.452 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../08-linux-image-generic-hwe-20.04_5.13.0.39.44~20.04.24_amd64.deb ... 2022-04-02T02:59:49.487 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking linux-image-generic-hwe-20.04 (5.13.0.39.44~20.04.24) ... 2022-04-02T02:59:49.789 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package libevdev2:amd64. 2022-04-02T02:59:49.805 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../09-libevdev2_1.9.0+dfsg-1ubuntu0.1_amd64.deb ... 2022-04-02T02:59:49.847 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking libevdev2:amd64 (1.9.0+dfsg-1ubuntu0.1) ... 2022-04-02T02:59:50.176 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package thermald. 2022-04-02T02:59:50.190 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../10-thermald_1.9.1-1ubuntu0.6_amd64.deb ... 2022-04-02T02:59:50.234 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking thermald (1.9.1-1ubuntu0.6) ... 2022-04-02T02:59:50.651 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package linux-image-generic-hwe-20.04. 2022-04-02T02:59:50.669 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../08-linux-image-generic-hwe-20.04_5.13.0.39.44~20.04.24_amd64.deb ... 2022-04-02T02:59:50.701 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking linux-image-generic-hwe-20.04 (5.13.0.39.44~20.04.24) ... 2022-04-02T02:59:50.791 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package upower. 2022-04-02T02:59:50.804 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../11-upower_0.99.11-1build2_amd64.deb ... 2022-04-02T02:59:50.859 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking upower (0.99.11-1build2) ... 2022-04-02T02:59:51.064 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package libevdev2:amd64. 2022-04-02T02:59:51.080 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../09-libevdev2_1.9.0+dfsg-1ubuntu0.1_amd64.deb ... 2022-04-02T02:59:51.120 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking libevdev2:amd64 (1.9.0+dfsg-1ubuntu0.1) ... 2022-04-02T02:59:51.266 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package usbmuxd. 2022-04-02T02:59:51.282 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../12-usbmuxd_1.1.1~git20191130.9af2b12-1_amd64.deb ... 2022-04-02T02:59:51.315 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking usbmuxd (1.1.1~git20191130.9af2b12-1) ... 2022-04-02T02:59:51.517 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package thermald. 2022-04-02T02:59:51.531 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../10-thermald_1.9.1-1ubuntu0.6_amd64.deb ... 2022-04-02T02:59:51.575 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking thermald (1.9.1-1ubuntu0.6) ... 2022-04-02T02:59:51.597 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up libplist3:amd64 (2.1.0-4build2) ... 2022-04-02T02:59:51.720 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up libdbus-glib-1-2:amd64 (0.110-5fakssync1) ... 2022-04-02T02:59:51.846 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up libusbmuxd6:amd64 (2.0.1-2) ... 2022-04-02T02:59:51.972 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up libupower-glib3:amd64 (0.99.11-1build2) ... 2022-04-02T02:59:52.123 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up libimobiledevice6:amd64 (1.2.1~git20191129.9f79242-1build1) ... 2022-04-02T02:59:52.141 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package upower. 2022-04-02T02:59:52.155 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../11-upower_0.99.11-1build2_amd64.deb ... 2022-04-02T02:59:52.234 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking upower (0.99.11-1build2) ... 2022-04-02T02:59:52.249 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up libevdev2:amd64 (1.9.0+dfsg-1ubuntu0.1) ... 2022-04-02T02:59:52.375 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up upower (0.99.11-1build2) ... 2022-04-02T02:59:52.698 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package usbmuxd. 2022-04-02T02:59:52.714 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../12-usbmuxd_1.1.1~git20191130.9af2b12-1_amd64.deb ... 2022-04-02T02:59:52.756 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking usbmuxd (1.1.1~git20191130.9af2b12-1) ... 2022-04-02T02:59:52.806 INFO:teuthology.orchestra.run.smithi104.stdout:upower.service is a disabled or a static unit, not starting it. 2022-04-02T02:59:52.851 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up usbmuxd (1.1.1~git20191130.9af2b12-1) ... 2022-04-02T02:59:53.006 INFO:teuthology.orchestra.run.smithi104.stdout:Warning: The home dir /var/lib/usbmux you specified can't be accessed: No such file or directory 2022-04-02T02:59:53.015 INFO:teuthology.orchestra.run.smithi104.stdout:Adding system user `usbmux' (UID 114) ... 2022-04-02T02:59:53.015 INFO:teuthology.orchestra.run.smithi104.stdout:Adding new user `usbmux' (UID 114) with group `plugdev' ... 2022-04-02T02:59:53.061 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up libplist3:amd64 (2.1.0-4build2) ... 2022-04-02T02:59:53.077 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package linux-image-generic-hwe-20.04. 2022-04-02T02:59:53.093 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../08-linux-image-generic-hwe-20.04_5.13.0.39.44~20.04.24_amd64.deb ... 2022-04-02T02:59:53.135 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking linux-image-generic-hwe-20.04 (5.13.0.39.44~20.04.24) ... 2022-04-02T02:59:53.202 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up libdbus-glib-1-2:amd64 (0.110-5fakssync1) ... 2022-04-02T02:59:53.328 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up libusbmuxd6:amd64 (2.0.1-2) ... 2022-04-02T02:59:53.479 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up libupower-glib3:amd64 (0.99.11-1build2) ... 2022-04-02T02:59:53.492 INFO:teuthology.orchestra.run.smithi104.stdout:Not creating home directory `/var/lib/usbmux'. 2022-04-02T02:59:53.501 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package libevdev2:amd64. 2022-04-02T02:59:53.514 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../09-libevdev2_1.9.0+dfsg-1ubuntu0.1_amd64.deb ... 2022-04-02T02:59:53.559 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking libevdev2:amd64 (1.9.0+dfsg-1ubuntu0.1) ... 2022-04-02T02:59:53.605 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up libimobiledevice6:amd64 (1.2.1~git20191129.9f79242-1build1) ... 2022-04-02T02:59:53.747 INFO:teuthology.orchestra.run.smithi104.stdout:usbmuxd.service is a disabled or a static unit, not starting it. 2022-04-02T02:59:53.756 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up libevdev2:amd64 (1.9.0+dfsg-1ubuntu0.1) ... 2022-04-02T02:59:53.789 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up thermald (1.9.1-1ubuntu0.6) ... 2022-04-02T02:59:53.883 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up upower (0.99.11-1build2) ... 2022-04-02T02:59:53.925 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package thermald. 2022-04-02T02:59:53.941 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../10-thermald_1.9.1-1ubuntu0.6_amd64.deb ... 2022-04-02T02:59:53.983 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking thermald (1.9.1-1ubuntu0.6) ... 2022-04-02T02:59:54.106 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.thermald.service -> /lib/systemd/system/thermald.service. 2022-04-02T02:59:54.106 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/thermald.service -> /lib/systemd/system/thermald.service. 2022-04-02T02:59:54.343 INFO:teuthology.orchestra.run.smithi142.stdout:upower.service is a disabled or a static unit, not starting it. 2022-04-02T02:59:54.392 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up usbmuxd (1.1.1~git20191130.9af2b12-1) ... 2022-04-02T02:59:54.531 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package upower. 2022-04-02T02:59:54.547 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../11-upower_0.99.11-1build2_amd64.deb ... 2022-04-02T02:59:54.589 INFO:teuthology.orchestra.run.smithi142.stdout:Warning: The home dir /var/lib/usbmux you specified can't be accessed: No such file or directory 2022-04-02T02:59:54.593 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up linux-modules-5.13.0-39-generic (5.13.0-39.44~20.04.1) ... 2022-04-02T02:59:54.600 INFO:teuthology.orchestra.run.smithi142.stdout:Adding system user `usbmux' (UID 114) ... 2022-04-02T02:59:54.601 INFO:teuthology.orchestra.run.smithi142.stdout:Adding new user `usbmux' (UID 114) with group `plugdev' ... 2022-04-02T02:59:54.602 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking upower (0.99.11-1build2) ... 2022-04-02T02:59:55.022 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package usbmuxd. 2022-04-02T02:59:55.037 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../12-usbmuxd_1.1.1~git20191130.9af2b12-1_amd64.deb ... 2022-04-02T02:59:55.075 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking usbmuxd (1.1.1~git20191130.9af2b12-1) ... 2022-04-02T02:59:55.118 INFO:teuthology.orchestra.run.smithi142.stdout:Not creating home directory `/var/lib/usbmux'. 2022-04-02T02:59:55.345 INFO:teuthology.orchestra.run.smithi142.stdout:usbmuxd.service is a disabled or a static unit, not starting it. 2022-04-02T02:59:55.389 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up libplist3:amd64 (2.1.0-4build2) ... 2022-04-02T02:59:55.414 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up thermald (1.9.1-1ubuntu0.6) ... 2022-04-02T02:59:55.513 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up libdbus-glib-1-2:amd64 (0.110-5fakssync1) ... 2022-04-02T02:59:55.673 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up libusbmuxd6:amd64 (2.0.1-2) ... 2022-04-02T02:59:55.722 INFO:teuthology.orchestra.run.smithi142.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.thermald.service -> /lib/systemd/system/thermald.service. 2022-04-02T02:59:55.722 INFO:teuthology.orchestra.run.smithi142.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/thermald.service -> /lib/systemd/system/thermald.service. 2022-04-02T02:59:55.800 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up libupower-glib3:amd64 (0.99.11-1build2) ... 2022-04-02T02:59:55.943 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up libimobiledevice6:amd64 (1.2.1~git20191129.9f79242-1build1) ... 2022-04-02T02:59:56.070 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up libevdev2:amd64 (1.9.0+dfsg-1ubuntu0.1) ... 2022-04-02T02:59:56.193 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up linux-modules-5.13.0-39-generic (5.13.0-39.44~20.04.1) ... 2022-04-02T02:59:56.199 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up upower (0.99.11-1build2) ... 2022-04-02T02:59:56.279 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up linux-image-5.13.0-39-generic (5.13.0-39.44~20.04.1) ... 2022-04-02T02:59:56.662 INFO:teuthology.orchestra.run.smithi029.stdout:upower.service is a disabled or a static unit, not starting it. 2022-04-02T02:59:56.710 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up usbmuxd (1.1.1~git20191130.9af2b12-1) ... 2022-04-02T02:59:56.919 INFO:teuthology.orchestra.run.smithi029.stdout:Warning: The home dir /var/lib/usbmux you specified can't be accessed: No such file or directory 2022-04-02T02:59:56.926 INFO:teuthology.orchestra.run.smithi029.stdout:Adding system user `usbmux' (UID 114) ... 2022-04-02T02:59:56.926 INFO:teuthology.orchestra.run.smithi029.stdout:Adding new user `usbmux' (UID 114) with group `plugdev' ... 2022-04-02T02:59:57.483 INFO:teuthology.orchestra.run.smithi029.stdout:Not creating home directory `/var/lib/usbmux'. 2022-04-02T02:59:57.571 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up linux-image-5.13.0-39-generic (5.13.0-39.44~20.04.1) ... 2022-04-02T02:59:57.604 INFO:teuthology.orchestra.run.smithi104.stdout:I: /boot/vmlinuz is now a symlink to vmlinuz-5.13.0-39-generic 2022-04-02T02:59:57.604 INFO:teuthology.orchestra.run.smithi104.stdout:I: /boot/initrd.img is now a symlink to initrd.img-5.13.0-39-generic 2022-04-02T02:59:57.697 INFO:teuthology.orchestra.run.smithi029.stdout:usbmuxd.service is a disabled or a static unit, not starting it. 2022-04-02T02:59:57.737 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up thermald (1.9.1-1ubuntu0.6) ... 2022-04-02T02:59:57.738 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up linux-modules-extra-5.13.0-39-generic (5.13.0-39.44~20.04.1) ... 2022-04-02T02:59:58.050 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.thermald.service -> /lib/systemd/system/thermald.service. 2022-04-02T02:59:58.051 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/thermald.service -> /lib/systemd/system/thermald.service. 2022-04-02T02:59:58.520 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up linux-modules-5.13.0-39-generic (5.13.0-39.44~20.04.1) ... 2022-04-02T02:59:58.865 INFO:teuthology.orchestra.run.smithi142.stdout:I: /boot/vmlinuz is now a symlink to vmlinuz-5.13.0-39-generic 2022-04-02T02:59:58.866 INFO:teuthology.orchestra.run.smithi142.stdout:I: /boot/initrd.img is now a symlink to initrd.img-5.13.0-39-generic 2022-04-02T02:59:59.031 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up linux-modules-extra-5.13.0-39-generic (5.13.0-39.44~20.04.1) ... 2022-04-02T02:59:59.080 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up linux-image-generic-hwe-20.04 (5.13.0.39.44~20.04.24) ... 2022-04-02T02:59:59.223 INFO:teuthology.orchestra.run.smithi104.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.2) ... 2022-04-02T02:59:59.927 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up linux-image-5.13.0-39-generic (5.13.0-39.44~20.04.1) ... 2022-04-02T03:00:00.782 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up linux-image-generic-hwe-20.04 (5.13.0.39.44~20.04.24) ... 2022-04-02T03:00:01.256 INFO:teuthology.orchestra.run.smithi029.stdout:I: /boot/vmlinuz is now a symlink to vmlinuz-5.13.0-39-generic 2022-04-02T03:00:01.257 INFO:teuthology.orchestra.run.smithi029.stdout:I: /boot/initrd.img is now a symlink to initrd.img-5.13.0-39-generic 2022-04-02T03:00:01.258 INFO:teuthology.orchestra.run.smithi142.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.2) ... 2022-04-02T03:00:01.399 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up linux-modules-extra-5.13.0-39-generic (5.13.0-39.44~20.04.1) ... 2022-04-02T03:00:02.759 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up linux-image-generic-hwe-20.04 (5.13.0.39.44~20.04.24) ... 2022-04-02T03:00:02.936 INFO:teuthology.orchestra.run.smithi029.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.2) ... 2022-04-02T03:00:03.303 INFO:teuthology.orchestra.run.smithi104.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-04-02T03:00:03.963 INFO:teuthology.orchestra.run.smithi104.stdout:Processing triggers for dbus (1.12.16-2ubuntu2.1) ... 2022-04-02T03:00:04.071 INFO:teuthology.orchestra.run.smithi104.stdout:Processing triggers for linux-image-5.13.0-39-generic (5.13.0-39.44~20.04.1) ... 2022-04-02T03:00:04.253 INFO:teuthology.orchestra.run.smithi104.stdout:/etc/kernel/postinst.d/initramfs-tools: 2022-04-02T03:00:04.253 INFO:teuthology.orchestra.run.smithi104.stdout:update-initramfs: Generating /boot/initrd.img-5.13.0-39-generic 2022-04-02T03:00:05.129 INFO:teuthology.orchestra.run.smithi142.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-04-02T03:00:05.815 INFO:teuthology.orchestra.run.smithi142.stdout:Processing triggers for dbus (1.12.16-2ubuntu2.1) ... 2022-04-02T03:00:05.915 INFO:teuthology.orchestra.run.smithi142.stdout:Processing triggers for linux-image-5.13.0-39-generic (5.13.0-39.44~20.04.1) ... 2022-04-02T03:00:06.178 INFO:teuthology.orchestra.run.smithi142.stdout:/etc/kernel/postinst.d/initramfs-tools: 2022-04-02T03:00:06.178 INFO:teuthology.orchestra.run.smithi142.stdout:update-initramfs: Generating /boot/initrd.img-5.13.0-39-generic 2022-04-02T03:00:09.495 INFO:teuthology.orchestra.run.smithi029.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-04-02T03:00:11.358 INFO:teuthology.orchestra.run.smithi029.stdout:Processing triggers for dbus (1.12.16-2ubuntu2.1) ... 2022-04-02T03:00:11.619 INFO:teuthology.orchestra.run.smithi029.stdout:Processing triggers for linux-image-5.13.0-39-generic (5.13.0-39.44~20.04.1) ... 2022-04-02T03:00:11.830 INFO:teuthology.orchestra.run.smithi029.stdout:/etc/kernel/postinst.d/initramfs-tools: 2022-04-02T03:00:11.830 INFO:teuthology.orchestra.run.smithi029.stdout:update-initramfs: Generating /boot/initrd.img-5.13.0-39-generic 2022-04-02T03:00:27.616 INFO:teuthology.orchestra.run.smithi104.stdout:/etc/kernel/postinst.d/zz-update-grub: 2022-04-02T03:00:27.617 INFO:teuthology.orchestra.run.smithi104.stdout:Sourcing file `/etc/default/grub' 2022-04-02T03:00:27.634 INFO:teuthology.orchestra.run.smithi104.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-04-02T03:00:27.654 INFO:teuthology.orchestra.run.smithi104.stdout:Generating grub configuration file ... 2022-04-02T03:00:27.942 INFO:teuthology.orchestra.run.smithi104.stdout:Found linux image: /boot/vmlinuz-5.13.0-39-generic 2022-04-02T03:00:27.979 INFO:teuthology.orchestra.run.smithi104.stdout:Found initrd image: /boot/initrd.img-5.13.0-39-generic 2022-04-02T03:00:28.361 INFO:teuthology.orchestra.run.smithi104.stdout:Found linux image: /boot/vmlinuz-5.4.0-97-generic 2022-04-02T03:00:28.373 INFO:teuthology.orchestra.run.smithi104.stdout:Found initrd image: /boot/initrd.img-5.4.0-97-generic 2022-04-02T03:00:30.223 INFO:teuthology.orchestra.run.smithi142.stdout:/etc/kernel/postinst.d/zz-update-grub: 2022-04-02T03:00:30.223 INFO:teuthology.orchestra.run.smithi142.stdout:Sourcing file `/etc/default/grub' 2022-04-02T03:00:30.235 INFO:teuthology.orchestra.run.smithi142.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-04-02T03:00:30.261 INFO:teuthology.orchestra.run.smithi142.stdout:Generating grub configuration file ... 2022-04-02T03:00:30.617 INFO:teuthology.orchestra.run.smithi142.stdout:Found linux image: /boot/vmlinuz-5.13.0-39-generic 2022-04-02T03:00:30.648 INFO:teuthology.orchestra.run.smithi142.stdout:Found initrd image: /boot/initrd.img-5.13.0-39-generic 2022-04-02T03:00:31.049 INFO:teuthology.orchestra.run.smithi142.stdout:Found linux image: /boot/vmlinuz-5.4.0-97-generic 2022-04-02T03:00:31.054 INFO:teuthology.orchestra.run.smithi104.stdout:done 2022-04-02T03:00:31.060 INFO:teuthology.orchestra.run.smithi142.stdout:Found initrd image: /boot/initrd.img-5.4.0-97-generic 2022-04-02T03:00:31.402 DEBUG:teuthology.orchestra.run.smithi104:> dpkg -s linux-image-generic-hwe-20.04 2022-04-02T03:00:31.429 INFO:teuthology.orchestra.run.smithi104.stdout:Package: linux-image-generic-hwe-20.04 2022-04-02T03:00:31.429 INFO:teuthology.orchestra.run.smithi104.stdout:Status: install ok installed 2022-04-02T03:00:31.429 INFO:teuthology.orchestra.run.smithi104.stdout:Priority: optional 2022-04-02T03:00:31.430 INFO:teuthology.orchestra.run.smithi104.stdout:Section: kernel 2022-04-02T03:00:31.430 INFO:teuthology.orchestra.run.smithi104.stdout:Installed-Size: 19 2022-04-02T03:00:31.430 INFO:teuthology.orchestra.run.smithi104.stdout:Maintainer: Ubuntu Kernel Team 2022-04-02T03:00:31.430 INFO:teuthology.orchestra.run.smithi104.stdout:Architecture: amd64 2022-04-02T03:00:31.431 INFO:teuthology.orchestra.run.smithi104.stdout:Source: linux-meta-hwe-5.13 2022-04-02T03:00:31.431 INFO:teuthology.orchestra.run.smithi104.stdout:Version: 5.13.0.39.44~20.04.24 2022-04-02T03:00:31.431 INFO:teuthology.orchestra.run.smithi104.stdout:Provides: virtualbox-guest-modules (= 5.13.0-39), wireguard-modules (= 1.0.0), zfs-modules (= 2.0.6-1ubuntu2.1) 2022-04-02T03:00:31.431 INFO:teuthology.orchestra.run.smithi104.stdout:Depends: linux-image-5.13.0-39-generic, linux-modules-extra-5.13.0-39-generic, linux-firmware, intel-microcode, amd64-microcode 2022-04-02T03:00:31.432 INFO:teuthology.orchestra.run.smithi104.stdout:Recommends: thermald 2022-04-02T03:00:31.432 INFO:teuthology.orchestra.run.smithi104.stdout:Description: Generic Linux kernel image 2022-04-02T03:00:31.432 INFO:teuthology.orchestra.run.smithi104.stdout: This package will always depend on the latest generic kernel image 2022-04-02T03:00:31.432 INFO:teuthology.orchestra.run.smithi104.stdout: available. 2022-04-02T03:00:31.433 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-97-generic Expected: 5.13.0-39-generic 2022-04-02T03:00:31.433 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2022-04-02T03:00:31.433 DEBUG:teuthology.task.kernel:src is distro, skipping download 2022-04-02T03:00:31.434 INFO:teuthology.task.kernel:Installing distro kernel on host.b... 2022-04-02T03:00:31.434 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi104.front.sepia.ceph.com, path=None, version=distro) 2022-04-02T03:00:31.434 DEBUG:teuthology.orchestra.run.smithi104:> sudo apt-get clean 2022-04-02T03:00:31.549 DEBUG:teuthology.orchestra.run.smithi104:> sudo apt-get update 2022-04-02T03:00:31.703 INFO:teuthology.orchestra.run.smithi104.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-04-02T03:00:31.704 INFO:teuthology.orchestra.run.smithi104.stdout:Hit:2 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-04-02T03:00:31.719 INFO:teuthology.orchestra.run.smithi104.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-04-02T03:00:31.749 INFO:teuthology.orchestra.run.smithi104.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-04-02T03:00:33.237 INFO:teuthology.orchestra.run.smithi104.stdout:Reading package lists... 2022-04-02T03:00:33.258 DEBUG:teuthology.orchestra.run.smithi104:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic-hwe-20.04 2022-04-02T03:00:33.324 INFO:teuthology.orchestra.run.smithi104.stdout:Reading package lists... 2022-04-02T03:00:33.498 INFO:teuthology.orchestra.run.smithi104.stdout:Building dependency tree... 2022-04-02T03:00:33.498 INFO:teuthology.orchestra.run.smithi104.stdout:Reading state information... 2022-04-02T03:00:33.679 INFO:teuthology.orchestra.run.smithi104.stdout:linux-image-generic-hwe-20.04 is already the newest version (5.13.0.39.44~20.04.24). 2022-04-02T03:00:33.679 INFO:teuthology.orchestra.run.smithi104.stdout:0 upgraded, 0 newly installed, 0 to remove and 120 not upgraded. 2022-04-02T03:00:33.681 DEBUG:teuthology.orchestra.run.smithi104:> dpkg -s linux-image-generic-hwe-20.04 2022-04-02T03:00:33.701 INFO:teuthology.orchestra.run.smithi104.stdout:Package: linux-image-generic-hwe-20.04 2022-04-02T03:00:33.702 INFO:teuthology.orchestra.run.smithi104.stdout:Status: install ok installed 2022-04-02T03:00:33.702 INFO:teuthology.orchestra.run.smithi104.stdout:Priority: optional 2022-04-02T03:00:33.702 INFO:teuthology.orchestra.run.smithi104.stdout:Section: kernel 2022-04-02T03:00:33.702 INFO:teuthology.orchestra.run.smithi104.stdout:Installed-Size: 19 2022-04-02T03:00:33.703 INFO:teuthology.orchestra.run.smithi104.stdout:Maintainer: Ubuntu Kernel Team 2022-04-02T03:00:33.703 INFO:teuthology.orchestra.run.smithi104.stdout:Architecture: amd64 2022-04-02T03:00:33.703 INFO:teuthology.orchestra.run.smithi104.stdout:Source: linux-meta-hwe-5.13 2022-04-02T03:00:33.703 INFO:teuthology.orchestra.run.smithi104.stdout:Version: 5.13.0.39.44~20.04.24 2022-04-02T03:00:33.704 INFO:teuthology.orchestra.run.smithi104.stdout:Provides: virtualbox-guest-modules (= 5.13.0-39), wireguard-modules (= 1.0.0), zfs-modules (= 2.0.6-1ubuntu2.1) 2022-04-02T03:00:33.704 INFO:teuthology.orchestra.run.smithi104.stdout:Depends: linux-image-5.13.0-39-generic, linux-modules-extra-5.13.0-39-generic, linux-firmware, intel-microcode, amd64-microcode 2022-04-02T03:00:33.704 INFO:teuthology.orchestra.run.smithi104.stdout:Recommends: thermald 2022-04-02T03:00:33.704 INFO:teuthology.orchestra.run.smithi104.stdout:Description: Generic Linux kernel image 2022-04-02T03:00:33.704 INFO:teuthology.orchestra.run.smithi104.stdout: This package will always depend on the latest generic kernel image 2022-04-02T03:00:33.705 INFO:teuthology.orchestra.run.smithi104.stdout: available. 2022-04-02T03:00:33.705 DEBUG:teuthology.orchestra.run.smithi104:> mktemp 2022-04-02T03:00:33.751 INFO:teuthology.orchestra.run.smithi104.stdout:/tmp/tmp.HcrT0ppCeq 2022-04-02T03:00:33.752 DEBUG:teuthology.orchestra.run.smithi104:> sudo cp /boot/grub/grub.cfg /tmp/tmp.HcrT0ppCeq 2022-04-02T03:00:33.767 DEBUG:teuthology.orchestra.run.smithi104:> sudo chmod 0666 /tmp/tmp.HcrT0ppCeq 2022-04-02T03:00:33.770 INFO:teuthology.orchestra.run.smithi142.stdout:done 2022-04-02T03:00:33.848 DEBUG:teuthology.orchestra.remote:smithi104:/tmp/tmp.HcrT0ppCeq is 8KB 2022-04-02T03:00:33.862 DEBUG:teuthology.orchestra.run.smithi104:> rm -fr /tmp/tmp.HcrT0ppCeq 2022-04-02T03:00:33.868 DEBUG:teuthology.orchestra.run.smithi104:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2022-04-02T03:00:33.925 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2022-04-02T03:00:33.925 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/etc/grub.d/01_ceph_kernel 2022-04-02T03:00:33.926 DEBUG:teuthology.orchestra.run.smithi104:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2022-04-02T03:00:33.971 INFO:teuthology.task.kernel:Distro Kernel Version: 5.13.0-39-generic 2022-04-02T03:00:33.972 DEBUG:teuthology.orchestra.run.smithi104:> sudo update-grub 2022-04-02T03:00:34.192 DEBUG:teuthology.orchestra.run.smithi142:> dpkg -s linux-image-generic-hwe-20.04 2022-04-02T03:00:34.219 INFO:teuthology.orchestra.run.smithi142.stdout:Package: linux-image-generic-hwe-20.04 2022-04-02T03:00:34.219 INFO:teuthology.orchestra.run.smithi142.stdout:Status: install ok installed 2022-04-02T03:00:34.219 INFO:teuthology.orchestra.run.smithi142.stdout:Priority: optional 2022-04-02T03:00:34.219 INFO:teuthology.orchestra.run.smithi142.stdout:Section: kernel 2022-04-02T03:00:34.220 INFO:teuthology.orchestra.run.smithi142.stdout:Installed-Size: 19 2022-04-02T03:00:34.220 INFO:teuthology.orchestra.run.smithi142.stdout:Maintainer: Ubuntu Kernel Team 2022-04-02T03:00:34.220 INFO:teuthology.orchestra.run.smithi142.stdout:Architecture: amd64 2022-04-02T03:00:34.220 INFO:teuthology.orchestra.run.smithi142.stdout:Source: linux-meta-hwe-5.13 2022-04-02T03:00:34.221 INFO:teuthology.orchestra.run.smithi142.stdout:Version: 5.13.0.39.44~20.04.24 2022-04-02T03:00:34.221 INFO:teuthology.orchestra.run.smithi142.stdout:Provides: virtualbox-guest-modules (= 5.13.0-39), wireguard-modules (= 1.0.0), zfs-modules (= 2.0.6-1ubuntu2.1) 2022-04-02T03:00:34.221 INFO:teuthology.orchestra.run.smithi142.stdout:Depends: linux-image-5.13.0-39-generic, linux-modules-extra-5.13.0-39-generic, linux-firmware, intel-microcode, amd64-microcode 2022-04-02T03:00:34.221 INFO:teuthology.orchestra.run.smithi142.stdout:Recommends: thermald 2022-04-02T03:00:34.222 INFO:teuthology.orchestra.run.smithi142.stdout:Description: Generic Linux kernel image 2022-04-02T03:00:34.222 INFO:teuthology.orchestra.run.smithi142.stdout: This package will always depend on the latest generic kernel image 2022-04-02T03:00:34.222 INFO:teuthology.orchestra.run.smithi142.stdout: available. 2022-04-02T03:00:34.222 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-97-generic Expected: 5.13.0-39-generic 2022-04-02T03:00:34.223 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2022-04-02T03:00:34.223 DEBUG:teuthology.task.kernel:src is distro, skipping download 2022-04-02T03:00:34.223 INFO:teuthology.task.kernel:Installing distro kernel on host.c... 2022-04-02T03:00:34.223 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi142.front.sepia.ceph.com, path=None, version=distro) 2022-04-02T03:00:34.224 DEBUG:teuthology.orchestra.run.smithi142:> sudo apt-get clean 2022-04-02T03:00:34.324 DEBUG:teuthology.orchestra.run.smithi142:> sudo apt-get update 2022-04-02T03:00:34.470 INFO:teuthology.orchestra.run.smithi142.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-04-02T03:00:34.470 INFO:teuthology.orchestra.run.smithi142.stdout:Hit:2 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-04-02T03:00:34.480 INFO:teuthology.orchestra.run.smithi142.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-04-02T03:00:34.500 INFO:teuthology.orchestra.run.smithi142.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-04-02T03:00:35.058 INFO:teuthology.orchestra.run.smithi104.stderr:Sourcing file `/etc/default/grub' 2022-04-02T03:00:35.060 INFO:teuthology.orchestra.run.smithi104.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-04-02T03:00:35.077 INFO:teuthology.orchestra.run.smithi104.stderr:Generating grub configuration file ... 2022-04-02T03:00:35.322 INFO:teuthology.orchestra.run.smithi104.stderr:Found linux image: /boot/vmlinuz-5.13.0-39-generic 2022-04-02T03:00:35.333 INFO:teuthology.orchestra.run.smithi104.stderr:Found initrd image: /boot/initrd.img-5.13.0-39-generic 2022-04-02T03:00:35.681 INFO:teuthology.orchestra.run.smithi104.stderr:Found linux image: /boot/vmlinuz-5.4.0-97-generic 2022-04-02T03:00:35.693 INFO:teuthology.orchestra.run.smithi104.stderr:Found initrd image: /boot/initrd.img-5.4.0-97-generic 2022-04-02T03:00:36.026 INFO:teuthology.orchestra.run.smithi142.stdout:Reading package lists... 2022-04-02T03:00:36.049 DEBUG:teuthology.orchestra.run.smithi142:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic-hwe-20.04 2022-04-02T03:00:36.119 INFO:teuthology.orchestra.run.smithi142.stdout:Reading package lists... 2022-04-02T03:00:36.295 INFO:teuthology.orchestra.run.smithi142.stdout:Building dependency tree... 2022-04-02T03:00:36.295 INFO:teuthology.orchestra.run.smithi142.stdout:Reading state information... 2022-04-02T03:00:36.473 INFO:teuthology.orchestra.run.smithi142.stdout:linux-image-generic-hwe-20.04 is already the newest version (5.13.0.39.44~20.04.24). 2022-04-02T03:00:36.473 INFO:teuthology.orchestra.run.smithi142.stdout:0 upgraded, 0 newly installed, 0 to remove and 120 not upgraded. 2022-04-02T03:00:36.475 DEBUG:teuthology.orchestra.run.smithi142:> dpkg -s linux-image-generic-hwe-20.04 2022-04-02T03:00:36.497 INFO:teuthology.orchestra.run.smithi142.stdout:Package: linux-image-generic-hwe-20.04 2022-04-02T03:00:36.498 INFO:teuthology.orchestra.run.smithi142.stdout:Status: install ok installed 2022-04-02T03:00:36.498 INFO:teuthology.orchestra.run.smithi142.stdout:Priority: optional 2022-04-02T03:00:36.498 INFO:teuthology.orchestra.run.smithi142.stdout:Section: kernel 2022-04-02T03:00:36.499 INFO:teuthology.orchestra.run.smithi142.stdout:Installed-Size: 19 2022-04-02T03:00:36.499 INFO:teuthology.orchestra.run.smithi142.stdout:Maintainer: Ubuntu Kernel Team 2022-04-02T03:00:36.499 INFO:teuthology.orchestra.run.smithi142.stdout:Architecture: amd64 2022-04-02T03:00:36.499 INFO:teuthology.orchestra.run.smithi142.stdout:Source: linux-meta-hwe-5.13 2022-04-02T03:00:36.500 INFO:teuthology.orchestra.run.smithi142.stdout:Version: 5.13.0.39.44~20.04.24 2022-04-02T03:00:36.500 INFO:teuthology.orchestra.run.smithi142.stdout:Provides: virtualbox-guest-modules (= 5.13.0-39), wireguard-modules (= 1.0.0), zfs-modules (= 2.0.6-1ubuntu2.1) 2022-04-02T03:00:36.500 INFO:teuthology.orchestra.run.smithi142.stdout:Depends: linux-image-5.13.0-39-generic, linux-modules-extra-5.13.0-39-generic, linux-firmware, intel-microcode, amd64-microcode 2022-04-02T03:00:36.500 INFO:teuthology.orchestra.run.smithi142.stdout:Recommends: thermald 2022-04-02T03:00:36.500 INFO:teuthology.orchestra.run.smithi142.stdout:Description: Generic Linux kernel image 2022-04-02T03:00:36.501 INFO:teuthology.orchestra.run.smithi142.stdout: This package will always depend on the latest generic kernel image 2022-04-02T03:00:36.501 INFO:teuthology.orchestra.run.smithi142.stdout: available. 2022-04-02T03:00:36.501 DEBUG:teuthology.orchestra.run.smithi142:> mktemp 2022-04-02T03:00:36.508 INFO:teuthology.orchestra.run.smithi142.stdout:/tmp/tmp.wLi4Si3npX 2022-04-02T03:00:36.508 DEBUG:teuthology.orchestra.run.smithi142:> sudo cp /boot/grub/grub.cfg /tmp/tmp.wLi4Si3npX 2022-04-02T03:00:36.520 DEBUG:teuthology.orchestra.run.smithi142:> sudo chmod 0666 /tmp/tmp.wLi4Si3npX 2022-04-02T03:00:36.562 DEBUG:teuthology.orchestra.remote:smithi142:/tmp/tmp.wLi4Si3npX is 8KB 2022-04-02T03:00:36.576 DEBUG:teuthology.orchestra.run.smithi142:> rm -fr /tmp/tmp.wLi4Si3npX 2022-04-02T03:00:36.582 DEBUG:teuthology.orchestra.run.smithi142:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2022-04-02T03:00:36.638 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2022-04-02T03:00:36.639 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd of=/etc/grub.d/01_ceph_kernel 2022-04-02T03:00:36.639 DEBUG:teuthology.orchestra.run.smithi142:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2022-04-02T03:00:36.680 INFO:teuthology.task.kernel:Distro Kernel Version: 5.13.0-39-generic 2022-04-02T03:00:36.681 DEBUG:teuthology.orchestra.run.smithi142:> sudo update-grub 2022-04-02T03:00:37.053 INFO:teuthology.orchestra.run.smithi029.stdout:/etc/kernel/postinst.d/zz-update-grub: 2022-04-02T03:00:37.053 INFO:teuthology.orchestra.run.smithi029.stdout:Sourcing file `/etc/default/grub' 2022-04-02T03:00:37.076 INFO:teuthology.orchestra.run.smithi029.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-04-02T03:00:37.107 INFO:teuthology.orchestra.run.smithi029.stdout:Generating grub configuration file ... 2022-04-02T03:00:37.320 INFO:teuthology.orchestra.run.smithi104.stderr:done 2022-04-02T03:00:37.322 DEBUG:teuthology.orchestra.run.smithi104:> sudo shutdown -r now 2022-04-02T03:00:37.491 INFO:teuthology.orchestra.run.smithi029.stdout:Found linux image: /boot/vmlinuz-5.13.0-39-generic 2022-04-02T03:00:37.526 INFO:teuthology.orchestra.run.smithi029.stdout:Found initrd image: /boot/initrd.img-5.13.0-39-generic 2022-04-02T03:00:37.908 INFO:teuthology.orchestra.run.smithi142.stderr:Sourcing file `/etc/default/grub' 2022-04-02T03:00:37.909 INFO:teuthology.orchestra.run.smithi029.stdout:Found linux image: /boot/vmlinuz-5.4.0-97-generic 2022-04-02T03:00:37.910 INFO:teuthology.orchestra.run.smithi142.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-04-02T03:00:37.919 INFO:teuthology.orchestra.run.smithi029.stdout:Found initrd image: /boot/initrd.img-5.4.0-97-generic 2022-04-02T03:00:37.926 INFO:teuthology.orchestra.run.smithi142.stderr:Generating grub configuration file ... 2022-04-02T03:00:38.200 INFO:teuthology.orchestra.run.smithi142.stderr:Found linux image: /boot/vmlinuz-5.13.0-39-generic 2022-04-02T03:00:38.212 INFO:teuthology.orchestra.run.smithi142.stderr:Found initrd image: /boot/initrd.img-5.13.0-39-generic 2022-04-02T03:00:38.547 INFO:teuthology.orchestra.run.smithi142.stderr:Found linux image: /boot/vmlinuz-5.4.0-97-generic 2022-04-02T03:00:38.559 INFO:teuthology.orchestra.run.smithi142.stderr:Found initrd image: /boot/initrd.img-5.4.0-97-generic 2022-04-02T03:00:40.137 INFO:teuthology.orchestra.run.smithi142.stderr:done 2022-04-02T03:00:40.139 DEBUG:teuthology.orchestra.run.smithi142:> sudo shutdown -r now 2022-04-02T03:00:40.882 INFO:teuthology.orchestra.run.smithi029.stdout:done 2022-04-02T03:00:41.328 DEBUG:teuthology.orchestra.run.smithi029:> dpkg -s linux-image-generic-hwe-20.04 2022-04-02T03:00:41.359 INFO:teuthology.orchestra.run.smithi029.stdout:Package: linux-image-generic-hwe-20.04 2022-04-02T03:00:41.359 INFO:teuthology.orchestra.run.smithi029.stdout:Status: install ok installed 2022-04-02T03:00:41.360 INFO:teuthology.orchestra.run.smithi029.stdout:Priority: optional 2022-04-02T03:00:41.360 INFO:teuthology.orchestra.run.smithi029.stdout:Section: kernel 2022-04-02T03:00:41.360 INFO:teuthology.orchestra.run.smithi029.stdout:Installed-Size: 19 2022-04-02T03:00:41.360 INFO:teuthology.orchestra.run.smithi029.stdout:Maintainer: Ubuntu Kernel Team 2022-04-02T03:00:41.361 INFO:teuthology.orchestra.run.smithi029.stdout:Architecture: amd64 2022-04-02T03:00:41.361 INFO:teuthology.orchestra.run.smithi029.stdout:Source: linux-meta-hwe-5.13 2022-04-02T03:00:41.361 INFO:teuthology.orchestra.run.smithi029.stdout:Version: 5.13.0.39.44~20.04.24 2022-04-02T03:00:41.361 INFO:teuthology.orchestra.run.smithi029.stdout:Provides: virtualbox-guest-modules (= 5.13.0-39), wireguard-modules (= 1.0.0), zfs-modules (= 2.0.6-1ubuntu2.1) 2022-04-02T03:00:41.362 INFO:teuthology.orchestra.run.smithi029.stdout:Depends: linux-image-5.13.0-39-generic, linux-modules-extra-5.13.0-39-generic, linux-firmware, intel-microcode, amd64-microcode 2022-04-02T03:00:41.362 INFO:teuthology.orchestra.run.smithi029.stdout:Recommends: thermald 2022-04-02T03:00:41.362 INFO:teuthology.orchestra.run.smithi029.stdout:Description: Generic Linux kernel image 2022-04-02T03:00:41.362 INFO:teuthology.orchestra.run.smithi029.stdout: This package will always depend on the latest generic kernel image 2022-04-02T03:00:41.363 INFO:teuthology.orchestra.run.smithi029.stdout: available. 2022-04-02T03:00:41.363 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-97-generic Expected: 5.13.0-39-generic 2022-04-02T03:00:41.363 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2022-04-02T03:00:41.364 DEBUG:teuthology.task.kernel:src is distro, skipping download 2022-04-02T03:00:41.364 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2022-04-02T03:00:41.364 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi029.front.sepia.ceph.com, path=None, version=distro) 2022-04-02T03:00:41.364 DEBUG:teuthology.orchestra.run.smithi029:> sudo apt-get clean 2022-04-02T03:00:41.435 DEBUG:teuthology.orchestra.run.smithi029:> sudo apt-get update 2022-04-02T03:00:41.595 INFO:teuthology.orchestra.run.smithi029.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-04-02T03:00:41.618 INFO:teuthology.orchestra.run.smithi029.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-04-02T03:00:41.662 INFO:teuthology.orchestra.run.smithi029.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-04-02T03:00:41.679 INFO:teuthology.orchestra.run.smithi029.stdout:Hit:4 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-04-02T03:00:43.211 INFO:teuthology.orchestra.run.smithi029.stdout:Reading package lists... 2022-04-02T03:00:43.233 DEBUG:teuthology.orchestra.run.smithi029:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic-hwe-20.04 2022-04-02T03:00:43.306 INFO:teuthology.orchestra.run.smithi029.stdout:Reading package lists... 2022-04-02T03:00:43.466 INFO:teuthology.orchestra.run.smithi029.stdout:Building dependency tree... 2022-04-02T03:00:43.467 INFO:teuthology.orchestra.run.smithi029.stdout:Reading state information... 2022-04-02T03:00:43.645 INFO:teuthology.orchestra.run.smithi029.stdout:linux-image-generic-hwe-20.04 is already the newest version (5.13.0.39.44~20.04.24). 2022-04-02T03:00:43.645 INFO:teuthology.orchestra.run.smithi029.stdout:0 upgraded, 0 newly installed, 0 to remove and 120 not upgraded. 2022-04-02T03:00:43.647 DEBUG:teuthology.orchestra.run.smithi029:> dpkg -s linux-image-generic-hwe-20.04 2022-04-02T03:00:43.667 INFO:teuthology.orchestra.run.smithi029.stdout:Package: linux-image-generic-hwe-20.04 2022-04-02T03:00:43.667 INFO:teuthology.orchestra.run.smithi029.stdout:Status: install ok installed 2022-04-02T03:00:43.667 INFO:teuthology.orchestra.run.smithi029.stdout:Priority: optional 2022-04-02T03:00:43.667 INFO:teuthology.orchestra.run.smithi029.stdout:Section: kernel 2022-04-02T03:00:43.668 INFO:teuthology.orchestra.run.smithi029.stdout:Installed-Size: 19 2022-04-02T03:00:43.668 INFO:teuthology.orchestra.run.smithi029.stdout:Maintainer: Ubuntu Kernel Team 2022-04-02T03:00:43.668 INFO:teuthology.orchestra.run.smithi029.stdout:Architecture: amd64 2022-04-02T03:00:43.669 INFO:teuthology.orchestra.run.smithi029.stdout:Source: linux-meta-hwe-5.13 2022-04-02T03:00:43.669 INFO:teuthology.orchestra.run.smithi029.stdout:Version: 5.13.0.39.44~20.04.24 2022-04-02T03:00:43.669 INFO:teuthology.orchestra.run.smithi029.stdout:Provides: virtualbox-guest-modules (= 5.13.0-39), wireguard-modules (= 1.0.0), zfs-modules (= 2.0.6-1ubuntu2.1) 2022-04-02T03:00:43.669 INFO:teuthology.orchestra.run.smithi029.stdout:Depends: linux-image-5.13.0-39-generic, linux-modules-extra-5.13.0-39-generic, linux-firmware, intel-microcode, amd64-microcode 2022-04-02T03:00:43.670 INFO:teuthology.orchestra.run.smithi029.stdout:Recommends: thermald 2022-04-02T03:00:43.670 INFO:teuthology.orchestra.run.smithi029.stdout:Description: Generic Linux kernel image 2022-04-02T03:00:43.670 INFO:teuthology.orchestra.run.smithi029.stdout: This package will always depend on the latest generic kernel image 2022-04-02T03:00:43.670 INFO:teuthology.orchestra.run.smithi029.stdout: available. 2022-04-02T03:00:43.671 DEBUG:teuthology.orchestra.run.smithi029:> mktemp 2022-04-02T03:00:43.716 INFO:teuthology.orchestra.run.smithi029.stdout:/tmp/tmp.shBEKWBBt0 2022-04-02T03:00:43.717 DEBUG:teuthology.orchestra.run.smithi029:> sudo cp /boot/grub/grub.cfg /tmp/tmp.shBEKWBBt0 2022-04-02T03:00:43.732 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 0666 /tmp/tmp.shBEKWBBt0 2022-04-02T03:00:43.824 DEBUG:teuthology.orchestra.remote:smithi029:/tmp/tmp.shBEKWBBt0 is 8KB 2022-04-02T03:00:43.838 DEBUG:teuthology.orchestra.run.smithi029:> rm -fr /tmp/tmp.shBEKWBBt0 2022-04-02T03:00:43.846 DEBUG:teuthology.orchestra.run.smithi029:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2022-04-02T03:00:43.901 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2022-04-02T03:00:43.902 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/etc/grub.d/01_ceph_kernel 2022-04-02T03:00:43.902 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2022-04-02T03:00:43.952 INFO:teuthology.task.kernel:Distro Kernel Version: 5.13.0-39-generic 2022-04-02T03:00:43.953 DEBUG:teuthology.orchestra.run.smithi029:> sudo update-grub 2022-04-02T03:00:45.936 INFO:teuthology.orchestra.run.smithi029.stderr:Sourcing file `/etc/default/grub' 2022-04-02T03:00:45.943 INFO:teuthology.orchestra.run.smithi029.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-04-02T03:00:45.964 INFO:teuthology.orchestra.run.smithi029.stderr:Generating grub configuration file ... 2022-04-02T03:00:46.293 INFO:teuthology.orchestra.run.smithi029.stderr:Found linux image: /boot/vmlinuz-5.13.0-39-generic 2022-04-02T03:00:46.304 INFO:teuthology.orchestra.run.smithi029.stderr:Found initrd image: /boot/initrd.img-5.13.0-39-generic 2022-04-02T03:00:46.662 INFO:teuthology.orchestra.run.smithi029.stderr:Found linux image: /boot/vmlinuz-5.4.0-97-generic 2022-04-02T03:00:46.674 INFO:teuthology.orchestra.run.smithi029.stderr:Found initrd image: /boot/initrd.img-5.4.0-97-generic 2022-04-02T03:00:48.277 INFO:teuthology.orchestra.run.smithi029.stderr:done 2022-04-02T03:00:48.279 DEBUG:teuthology.orchestra.run.smithi029:> sudo shutdown -r now 2022-04-02T03:01:07.343 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2022-04-02T03:01:07.343 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi104.front.sepia.ceph.com' 2022-04-02T03:01:07.344 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-04-02T03:01:10.146 INFO:teuthology.task.kernel:Checking client host.c for new kernel version... 2022-04-02T03:01:10.146 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com' 2022-04-02T03:01:10.147 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi142.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-04-02T03:01:18.291 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2022-04-02T03:01:18.291 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi029.front.sepia.ceph.com' 2022-04-02T03:01:18.292 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi029.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-04-02T03:01:25.835 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.104 2022-04-02T03:01:28.651 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.142 2022-04-02T03:01:36.843 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.29 2022-04-02T03:01:55.854 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi104.front.sepia.ceph.com' 2022-04-02T03:01:55.856 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-04-02T03:01:58.654 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com' 2022-04-02T03:01:58.655 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi142.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-04-02T03:02:06.853 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi029.front.sepia.ceph.com' 2022-04-02T03:02:06.854 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi029.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-04-02T03:02:13.195 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.104 2022-04-02T03:02:17.039 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.142 2022-04-02T03:02:24.203 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.29 2022-04-02T03:02:43.202 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi104.front.sepia.ceph.com' 2022-04-02T03:02:43.204 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-04-02T03:02:43.486 DEBUG:teuthology.orchestra.run.smithi104:> true 2022-04-02T03:02:43.994 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi104.front.sepia.ceph.com' 2022-04-02T03:02:43.994 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "5.13.0-39-generic"... 2022-04-02T03:02:43.995 DEBUG:teuthology.orchestra.run.smithi104:> uname -r 2022-04-02T03:02:44.000 INFO:teuthology.orchestra.run.smithi104.stdout:5.13.0-39-generic 2022-04-02T03:02:44.001 DEBUG:teuthology.task.kernel:current kernel version is 5.13.0-39-generic vs 5.13.0-39-generic 2022-04-02T03:02:44.001 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2022-04-02T03:02:44.001 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2022-04-02T03:02:45.002 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2022-04-02T03:02:45.003 DEBUG:teuthology.orchestra.run.smithi104:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2022-04-02T03:02:45.370 INFO:teuthology.orchestra.run.smithi104.stdout:ttyS1 2022-04-02T03:02:45.389 DEBUG:teuthology.parallel:result is None 2022-04-02T03:02:47.040 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com' 2022-04-02T03:02:47.041 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi142.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-04-02T03:02:47.286 DEBUG:teuthology.orchestra.run.smithi142:> true 2022-04-02T03:02:47.773 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi142.front.sepia.ceph.com' 2022-04-02T03:02:47.773 INFO:teuthology.task.kernel:Checking kernel version of host.c, want "5.13.0-39-generic"... 2022-04-02T03:02:47.773 DEBUG:teuthology.orchestra.run.smithi142:> uname -r 2022-04-02T03:02:47.778 INFO:teuthology.orchestra.run.smithi142.stdout:5.13.0-39-generic 2022-04-02T03:02:47.778 DEBUG:teuthology.task.kernel:current kernel version is 5.13.0-39-generic vs 5.13.0-39-generic 2022-04-02T03:02:47.779 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2022-04-02T03:02:47.779 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2022-04-02T03:02:48.780 INFO:teuthology.task.kernel:Enabling kdb on host.c... 2022-04-02T03:02:48.780 DEBUG:teuthology.orchestra.run.smithi142:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2022-04-02T03:02:48.854 INFO:teuthology.orchestra.run.smithi142.stdout:ttyS1 2022-04-02T03:02:48.880 DEBUG:teuthology.parallel:result is None 2022-04-02T03:02:54.205 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi029.front.sepia.ceph.com' 2022-04-02T03:02:54.206 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi029.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-04-02T03:02:54.410 DEBUG:teuthology.orchestra.run.smithi029:> true 2022-04-02T03:02:54.993 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi029.front.sepia.ceph.com' 2022-04-02T03:02:54.994 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "5.13.0-39-generic"... 2022-04-02T03:02:54.994 DEBUG:teuthology.orchestra.run.smithi029:> uname -r 2022-04-02T03:02:55.043 INFO:teuthology.orchestra.run.smithi029.stdout:5.13.0-39-generic 2022-04-02T03:02:55.043 DEBUG:teuthology.task.kernel:current kernel version is 5.13.0-39-generic vs 5.13.0-39-generic 2022-04-02T03:02:55.044 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2022-04-02T03:02:55.044 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2022-04-02T03:02:56.046 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2022-04-02T03:02:56.047 DEBUG:teuthology.orchestra.run.smithi029:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2022-04-02T03:02:56.129 INFO:teuthology.orchestra.run.smithi029.stdout:ttyS1 2022-04-02T03:02:56.155 DEBUG:teuthology.parallel:result is None 2022-04-02T03:02:56.156 INFO:teuthology.run_tasks:Running task internal.base... 2022-04-02T03:02:56.166 INFO:teuthology.task.internal:Creating test directory... 2022-04-02T03:02:56.167 DEBUG:teuthology.orchestra.run.smithi029:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2022-04-02T03:02:56.169 DEBUG:teuthology.orchestra.run.smithi104:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2022-04-02T03:02:56.172 DEBUG:teuthology.orchestra.run.smithi142:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2022-04-02T03:02:56.180 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2022-04-02T03:02:56.253 INFO:teuthology.run_tasks:Running task internal.archive... 2022-04-02T03:02:56.371 INFO:teuthology.task.internal:Creating archive directory... 2022-04-02T03:02:56.736 DEBUG:teuthology.orchestra.run.smithi029:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2022-04-02T03:02:56.739 DEBUG:teuthology.orchestra.run.smithi104:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2022-04-02T03:02:56.741 DEBUG:teuthology.orchestra.run.smithi142:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2022-04-02T03:02:56.907 INFO:teuthology.run_tasks:Running task internal.coredump... 2022-04-02T03:02:56.919 INFO:teuthology.task.internal:Enabling coredump saving... 2022-04-02T03:02:56.919 DEBUG:teuthology.orchestra.run.smithi029:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2022-04-02T03:02:56.922 DEBUG:teuthology.orchestra.run.smithi104:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2022-04-02T03:02:56.925 DEBUG:teuthology.orchestra.run.smithi142:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2022-04-02T03:02:56.943 INFO:teuthology.orchestra.run.smithi029.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-04-02T03:02:56.949 INFO:teuthology.orchestra.run.smithi142.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-04-02T03:02:56.953 INFO:teuthology.orchestra.run.smithi029.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-04-02T03:02:56.959 INFO:teuthology.orchestra.run.smithi142.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-04-02T03:02:56.982 INFO:teuthology.orchestra.run.smithi104.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-04-02T03:02:56.994 INFO:teuthology.orchestra.run.smithi104.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-04-02T03:02:56.995 INFO:teuthology.run_tasks:Running task internal.sudo... 2022-04-02T03:02:57.004 INFO:teuthology.task.internal:Configuring sudo... 2022-04-02T03:02:57.005 DEBUG:teuthology.orchestra.run.smithi029:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2022-04-02T03:02:57.008 DEBUG:teuthology.orchestra.run.smithi104:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2022-04-02T03:02:57.011 DEBUG:teuthology.orchestra.run.smithi142:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2022-04-02T03:02:57.027 INFO:teuthology.run_tasks:Running task internal.syslog... 2022-04-02T03:02:57.059 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2022-04-02T03:02:57.059 DEBUG:teuthology.orchestra.run.smithi029:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2022-04-02T03:02:57.070 DEBUG:teuthology.orchestra.run.smithi104:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2022-04-02T03:02:57.073 DEBUG:teuthology.orchestra.run.smithi142:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2022-04-02T03:02:57.080 DEBUG:teuthology.orchestra.run.smithi029:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2022-04-02T03:02:57.086 DEBUG:teuthology.orchestra.run.smithi029:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2022-04-02T03:02:57.095 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2022-04-02T03:02:57.095 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2022-04-02T03:02:57.159 DEBUG:teuthology.orchestra.run.smithi104:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2022-04-02T03:02:57.167 DEBUG:teuthology.orchestra.run.smithi104:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2022-04-02T03:02:57.173 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2022-04-02T03:02:57.173 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2022-04-02T03:02:57.189 DEBUG:teuthology.orchestra.run.smithi142:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2022-04-02T03:02:57.197 DEBUG:teuthology.orchestra.run.smithi142:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2022-04-02T03:02:57.203 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2022-04-02T03:02:57.203 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2022-04-02T03:02:57.281 DEBUG:teuthology.orchestra.run.smithi029:> sudo service rsyslog restart 2022-04-02T03:02:57.284 DEBUG:teuthology.orchestra.run.smithi104:> sudo service rsyslog restart 2022-04-02T03:02:57.286 DEBUG:teuthology.orchestra.run.smithi142:> sudo service rsyslog restart 2022-04-02T03:02:57.332 INFO:teuthology.run_tasks:Running task internal.timer... 2022-04-02T03:02:57.340 INFO:teuthology.task.internal:Starting timer... 2022-04-02T03:02:57.341 INFO:teuthology.run_tasks:Running task pcp... 2022-04-02T03:02:57.372 INFO:teuthology.run_tasks:Running task selinux... 2022-04-02T03:02:57.404 DEBUG:teuthology.task.selinux:Excluding smithi029: OS 'ubuntu' does not support SELinux 2022-04-02T03:02:57.405 DEBUG:teuthology.task.selinux:Excluding smithi104: OS 'ubuntu' does not support SELinux 2022-04-02T03:02:57.405 DEBUG:teuthology.task.selinux:Excluding smithi142: OS 'ubuntu' does not support SELinux 2022-04-02T03:02:57.406 DEBUG:teuthology.task.selinux:Getting current SELinux state 2022-04-02T03:02:57.406 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2022-04-02T03:02:57.407 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2022-04-02T03:02:57.407 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2022-04-02T03:02:57.453 DEBUG:teuthology.repo_utils:Setting repo remote to git://git.ceph.com/git/ceph-cm-ansible.git 2022-04-02T03:02:57.459 INFO:teuthology.repo_utils:Fetching master from origin 2022-04-02T03:02:57.491 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master to origin/master 2022-04-02T03:02:57.510 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2022-04-02T03:02:57.512 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi029.front.sepia.ceph.com,smithi104.front.sepia.ceph.com,smithi142.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/cephlab.yml 2022-04-02T03:06:16.292 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi029.front.sepia.ceph.com'), Remote(name='ubuntu@smithi104.front.sepia.ceph.com'), Remote(name='ubuntu@smithi142.front.sepia.ceph.com')] 2022-04-02T03:06:16.296 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi029.front.sepia.ceph.com' 2022-04-02T03:06:16.298 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi029.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-04-02T03:06:16.349 DEBUG:teuthology.orchestra.run.smithi029:> true 2022-04-02T03:06:16.469 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi104.front.sepia.ceph.com' 2022-04-02T03:06:16.470 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-04-02T03:06:16.514 DEBUG:teuthology.orchestra.run.smithi104:> true 2022-04-02T03:06:16.636 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com' 2022-04-02T03:06:16.637 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi142.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-04-02T03:06:16.691 DEBUG:teuthology.orchestra.run.smithi142:> true 2022-04-02T03:06:16.813 INFO:teuthology.run_tasks:Running task clock... 2022-04-02T03:06:16.828 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2022-04-02T03:06:16.828 INFO:teuthology.orchestra.run:Running command with timeout 360 2022-04-02T03:06:16.830 DEBUG:teuthology.orchestra.run.smithi029:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-04-02T03:06:16.833 INFO:teuthology.orchestra.run:Running command with timeout 360 2022-04-02T03:06:16.834 DEBUG:teuthology.orchestra.run.smithi104:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-04-02T03:06:16.838 INFO:teuthology.orchestra.run:Running command with timeout 360 2022-04-02T03:06:16.838 DEBUG:teuthology.orchestra.run.smithi142:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-04-02T03:06:16.868 INFO:teuthology.orchestra.run.smithi029.stdout: 2 Apr 03:06:16 ntpd[9471]: ntpd 4.2.8p12@1.3728-o (1): Starting 2022-04-02T03:06:16.868 INFO:teuthology.orchestra.run.smithi029.stdout: 2 Apr 03:06:16 ntpd[9471]: Command line: ntpd -gq 2022-04-02T03:06:16.869 INFO:teuthology.orchestra.run.smithi029.stdout: 2 Apr 03:06:16 ntpd[9471]: proto: precision = 0.095 usec (-23) 2022-04-02T03:06:16.869 INFO:teuthology.orchestra.run.smithi029.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-04-02T03:06:16.869 INFO:teuthology.orchestra.run.smithi029.stdout: 2 Apr 03:06:16 ntpd[9471]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-04-02T03:06:16.870 INFO:teuthology.orchestra.run.smithi029.stderr:restrict ::: KOD does nothing without LIMITED. 2022-04-02T03:06:16.872 INFO:teuthology.orchestra.run.smithi029.stdout: 2 Apr 03:06:16 ntpd[9471]: restrict ::: KOD does nothing without LIMITED. 2022-04-02T03:06:16.872 INFO:teuthology.orchestra.run.smithi029.stdout: 2 Apr 03:06:16 ntpd[9471]: Listen and drop on 0 v6wildcard [::]:123 2022-04-02T03:06:16.872 INFO:teuthology.orchestra.run.smithi029.stdout: 2 Apr 03:06:16 ntpd[9471]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2022-04-02T03:06:16.872 INFO:teuthology.orchestra.run.smithi029.stdout: 2 Apr 03:06:16 ntpd[9471]: Listen normally on 2 lo 127.0.0.1:123 2022-04-02T03:06:16.873 INFO:teuthology.orchestra.run.smithi029.stdout: 2 Apr 03:06:16 ntpd[9471]: Listen normally on 3 ens1f0 172.21.15.29:123 2022-04-02T03:06:16.873 INFO:teuthology.orchestra.run.smithi029.stdout: 2 Apr 03:06:16 ntpd[9471]: Listen normally on 4 lo [::1]:123 2022-04-02T03:06:16.873 INFO:teuthology.orchestra.run.smithi029.stdout: 2 Apr 03:06:16 ntpd[9471]: Listen normally on 5 ens1f0 [fe80::ec4:7aff:febd:125a%4]:123 2022-04-02T03:06:16.873 INFO:teuthology.orchestra.run.smithi029.stdout: 2 Apr 03:06:16 ntpd[9471]: Listening on routing socket on fd #22 for interface updates 2022-04-02T03:06:16.874 INFO:teuthology.orchestra.run.smithi104.stdout: 2 Apr 03:06:16 ntpd[9536]: ntpd 4.2.8p12@1.3728-o (1): Starting 2022-04-02T03:06:16.874 INFO:teuthology.orchestra.run.smithi104.stdout: 2 Apr 03:06:16 ntpd[9536]: Command line: ntpd -gq 2022-04-02T03:06:16.875 INFO:teuthology.orchestra.run.smithi104.stdout: 2 Apr 03:06:16 ntpd[9536]: proto: precision = 0.095 usec (-23) 2022-04-02T03:06:16.875 INFO:teuthology.orchestra.run.smithi104.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-04-02T03:06:16.875 INFO:teuthology.orchestra.run.smithi104.stdout: 2 Apr 03:06:16 ntpd[9536]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-04-02T03:06:16.876 INFO:teuthology.orchestra.run.smithi104.stderr:restrict ::: KOD does nothing without LIMITED. 2022-04-02T03:06:16.877 INFO:teuthology.orchestra.run.smithi104.stdout: 2 Apr 03:06:16 ntpd[9536]: restrict ::: KOD does nothing without LIMITED. 2022-04-02T03:06:16.877 INFO:teuthology.orchestra.run.smithi104.stdout: 2 Apr 03:06:16 ntpd[9536]: Listen and drop on 0 v6wildcard [::]:123 2022-04-02T03:06:16.878 INFO:teuthology.orchestra.run.smithi104.stdout: 2 Apr 03:06:16 ntpd[9536]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2022-04-02T03:06:16.878 INFO:teuthology.orchestra.run.smithi104.stdout: 2 Apr 03:06:16 ntpd[9536]: Listen normally on 2 lo 127.0.0.1:123 2022-04-02T03:06:16.878 INFO:teuthology.orchestra.run.smithi104.stdout: 2 Apr 03:06:16 ntpd[9536]: Listen normally on 3 enp3s0f1 172.21.15.104:123 2022-04-02T03:06:16.878 INFO:teuthology.orchestra.run.smithi104.stdout: 2 Apr 03:06:16 ntpd[9536]: Listen normally on 4 lo [::1]:123 2022-04-02T03:06:16.878 INFO:teuthology.orchestra.run.smithi104.stdout: 2 Apr 03:06:16 ntpd[9536]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:ce89%5]:123 2022-04-02T03:06:16.879 INFO:teuthology.orchestra.run.smithi104.stdout: 2 Apr 03:06:16 ntpd[9536]: Listening on routing socket on fd #22 for interface updates 2022-04-02T03:06:16.879 INFO:teuthology.orchestra.run.smithi142.stdout: 2 Apr 03:06:16 ntpd[9599]: ntpd 4.2.8p12@1.3728-o (1): Starting 2022-04-02T03:06:16.880 INFO:teuthology.orchestra.run.smithi142.stdout: 2 Apr 03:06:16 ntpd[9599]: Command line: ntpd -gq 2022-04-02T03:06:16.880 INFO:teuthology.orchestra.run.smithi142.stdout: 2 Apr 03:06:16 ntpd[9599]: proto: precision = 0.046 usec (-24) 2022-04-02T03:06:16.880 INFO:teuthology.orchestra.run.smithi142.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-04-02T03:06:16.881 INFO:teuthology.orchestra.run.smithi142.stdout: 2 Apr 03:06:16 ntpd[9599]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-04-02T03:06:16.881 INFO:teuthology.orchestra.run.smithi142.stderr:restrict ::: KOD does nothing without LIMITED. 2022-04-02T03:06:16.883 INFO:teuthology.orchestra.run.smithi142.stdout: 2 Apr 03:06:16 ntpd[9599]: restrict ::: KOD does nothing without LIMITED. 2022-04-02T03:06:16.883 INFO:teuthology.orchestra.run.smithi142.stdout: 2 Apr 03:06:16 ntpd[9599]: Listen and drop on 0 v6wildcard [::]:123 2022-04-02T03:06:16.883 INFO:teuthology.orchestra.run.smithi142.stdout: 2 Apr 03:06:16 ntpd[9599]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2022-04-02T03:06:16.883 INFO:teuthology.orchestra.run.smithi142.stdout: 2 Apr 03:06:16 ntpd[9599]: Listen normally on 2 lo 127.0.0.1:123 2022-04-02T03:06:16.884 INFO:teuthology.orchestra.run.smithi142.stdout: 2 Apr 03:06:16 ntpd[9599]: Listen normally on 3 enp3s0f1 172.21.15.142:123 2022-04-02T03:06:16.884 INFO:teuthology.orchestra.run.smithi142.stdout: 2 Apr 03:06:16 ntpd[9599]: Listen normally on 4 lo [::1]:123 2022-04-02T03:06:16.884 INFO:teuthology.orchestra.run.smithi142.stdout: 2 Apr 03:06:16 ntpd[9599]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:6e0d%5]:123 2022-04-02T03:06:16.884 INFO:teuthology.orchestra.run.smithi142.stdout: 2 Apr 03:06:16 ntpd[9599]: Listening on routing socket on fd #22 for interface updates 2022-04-02T03:06:17.869 INFO:teuthology.orchestra.run.smithi029.stderr: 2 Apr 03:06:17 ntpd[9471]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-02T03:06:17.869 INFO:teuthology.orchestra.run.smithi029.stderr: 2 Apr 03:06:17 ntpd[9471]: can't open /var/log/ntpstats/rawstats.20220402: Permission denied 2022-04-02T03:06:17.869 INFO:teuthology.orchestra.run.smithi029.stderr: 2 Apr 03:06:17 ntpd[9471]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-02T03:06:17.869 INFO:teuthology.orchestra.run.smithi029.stderr: 2 Apr 03:06:17 ntpd[9471]: can't open /var/log/ntpstats/rawstats.20220402: Permission denied 2022-04-02T03:06:17.870 INFO:teuthology.orchestra.run.smithi029.stderr: 2 Apr 03:06:17 ntpd[9471]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-02T03:06:17.870 INFO:teuthology.orchestra.run.smithi029.stderr: 2 Apr 03:06:17 ntpd[9471]: can't open /var/log/ntpstats/rawstats.20220402: Permission denied 2022-04-02T03:06:17.870 INFO:teuthology.orchestra.run.smithi029.stderr: 2 Apr 03:06:17 ntpd[9471]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-04-02T03:06:17.870 INFO:teuthology.orchestra.run.smithi029.stderr: 2 Apr 03:06:17 ntpd[9471]: can't open /var/log/ntpstats/peerstats.20220402: Permission denied 2022-04-02T03:06:17.871 INFO:teuthology.orchestra.run.smithi029.stderr: 2 Apr 03:06:17 ntpd[9471]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-02T03:06:17.871 INFO:teuthology.orchestra.run.smithi029.stderr: 2 Apr 03:06:17 ntpd[9471]: can't open /var/log/ntpstats/rawstats.20220402: Permission denied 2022-04-02T03:06:17.871 INFO:teuthology.orchestra.run.smithi029.stderr: 2 Apr 03:06:17 ntpd[9471]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-04-02T03:06:17.871 INFO:teuthology.orchestra.run.smithi029.stderr: 2 Apr 03:06:17 ntpd[9471]: can't open /var/log/ntpstats/peerstats.20220402: Permission denied 2022-04-02T03:06:17.872 INFO:teuthology.orchestra.run.smithi104.stderr: 2 Apr 03:06:17 ntpd[9536]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-02T03:06:17.873 INFO:teuthology.orchestra.run.smithi104.stderr: 2 Apr 03:06:17 ntpd[9536]: can't open /var/log/ntpstats/rawstats.20220402: Permission denied 2022-04-02T03:06:17.873 INFO:teuthology.orchestra.run.smithi104.stderr: 2 Apr 03:06:17 ntpd[9536]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-02T03:06:17.873 INFO:teuthology.orchestra.run.smithi104.stderr: 2 Apr 03:06:17 ntpd[9536]: can't open /var/log/ntpstats/rawstats.20220402: Permission denied 2022-04-02T03:06:17.873 INFO:teuthology.orchestra.run.smithi104.stderr: 2 Apr 03:06:17 ntpd[9536]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-04-02T03:06:17.874 INFO:teuthology.orchestra.run.smithi104.stderr: 2 Apr 03:06:17 ntpd[9536]: can't open /var/log/ntpstats/peerstats.20220402: Permission denied 2022-04-02T03:06:17.875 INFO:teuthology.orchestra.run.smithi142.stderr: 2 Apr 03:06:17 ntpd[9599]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-02T03:06:17.875 INFO:teuthology.orchestra.run.smithi142.stderr: 2 Apr 03:06:17 ntpd[9599]: can't open /var/log/ntpstats/rawstats.20220402: Permission denied 2022-04-02T03:06:17.875 INFO:teuthology.orchestra.run.smithi142.stderr: 2 Apr 03:06:17 ntpd[9599]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-02T03:06:17.875 INFO:teuthology.orchestra.run.smithi142.stderr: 2 Apr 03:06:17 ntpd[9599]: can't open /var/log/ntpstats/rawstats.20220402: Permission denied 2022-04-02T03:06:17.876 INFO:teuthology.orchestra.run.smithi142.stderr: 2 Apr 03:06:17 ntpd[9599]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-04-02T03:06:17.876 INFO:teuthology.orchestra.run.smithi142.stderr: 2 Apr 03:06:17 ntpd[9599]: can't open /var/log/ntpstats/peerstats.20220402: Permission denied 2022-04-02T03:06:18.868 INFO:teuthology.orchestra.run.smithi029.stderr: 2 Apr 03:06:18 ntpd[9471]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-02T03:06:18.868 INFO:teuthology.orchestra.run.smithi029.stderr: 2 Apr 03:06:18 ntpd[9471]: can't open /var/log/ntpstats/rawstats.20220402: Permission denied 2022-04-02T03:06:18.869 INFO:teuthology.orchestra.run.smithi029.stderr: 2 Apr 03:06:18 ntpd[9471]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-02T03:06:18.869 INFO:teuthology.orchestra.run.smithi029.stderr: 2 Apr 03:06:18 ntpd[9471]: can't open /var/log/ntpstats/rawstats.20220402: Permission denied 2022-04-02T03:06:18.869 INFO:teuthology.orchestra.run.smithi029.stderr: 2 Apr 03:06:18 ntpd[9471]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-04-02T03:06:18.869 INFO:teuthology.orchestra.run.smithi029.stderr: 2 Apr 03:06:18 ntpd[9471]: can't open /var/log/ntpstats/peerstats.20220402: Permission denied 2022-04-02T03:06:18.875 INFO:teuthology.orchestra.run.smithi142.stderr: 2 Apr 03:06:18 ntpd[9599]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-02T03:06:18.875 INFO:teuthology.orchestra.run.smithi142.stderr: 2 Apr 03:06:18 ntpd[9599]: can't open /var/log/ntpstats/rawstats.20220402: Permission denied 2022-04-02T03:06:18.875 INFO:teuthology.orchestra.run.smithi142.stderr: 2 Apr 03:06:18 ntpd[9599]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-02T03:06:18.875 INFO:teuthology.orchestra.run.smithi142.stderr: 2 Apr 03:06:18 ntpd[9599]: can't open /var/log/ntpstats/rawstats.20220402: Permission denied 2022-04-02T03:06:18.876 INFO:teuthology.orchestra.run.smithi142.stderr: 2 Apr 03:06:18 ntpd[9599]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-04-02T03:06:18.876 INFO:teuthology.orchestra.run.smithi142.stderr: 2 Apr 03:06:18 ntpd[9599]: can't open /var/log/ntpstats/peerstats.20220402: Permission denied 2022-04-02T03:06:19.869 INFO:teuthology.orchestra.run.smithi029.stderr: 2 Apr 03:06:19 ntpd[9471]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-02T03:06:19.869 INFO:teuthology.orchestra.run.smithi029.stderr: 2 Apr 03:06:19 ntpd[9471]: can't open /var/log/ntpstats/rawstats.20220402: Permission denied 2022-04-02T03:06:19.870 INFO:teuthology.orchestra.run.smithi029.stderr: 2 Apr 03:06:19 ntpd[9471]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-02T03:06:19.870 INFO:teuthology.orchestra.run.smithi029.stderr: 2 Apr 03:06:19 ntpd[9471]: can't open /var/log/ntpstats/rawstats.20220402: Permission denied 2022-04-02T03:06:19.870 INFO:teuthology.orchestra.run.smithi029.stderr: 2 Apr 03:06:19 ntpd[9471]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-02T03:06:19.870 INFO:teuthology.orchestra.run.smithi029.stderr: 2 Apr 03:06:19 ntpd[9471]: can't open /var/log/ntpstats/rawstats.20220402: Permission denied 2022-04-02T03:06:19.871 INFO:teuthology.orchestra.run.smithi029.stderr: 2 Apr 03:06:19 ntpd[9471]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-02T03:06:19.871 INFO:teuthology.orchestra.run.smithi029.stderr: 2 Apr 03:06:19 ntpd[9471]: can't open /var/log/ntpstats/rawstats.20220402: Permission denied 2022-04-02T03:06:19.871 INFO:teuthology.orchestra.run.smithi029.stderr: 2 Apr 03:06:19 ntpd[9471]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-04-02T03:06:19.871 INFO:teuthology.orchestra.run.smithi029.stderr: 2 Apr 03:06:19 ntpd[9471]: can't open /var/log/ntpstats/peerstats.20220402: Permission denied 2022-04-02T03:06:19.872 INFO:teuthology.orchestra.run.smithi029.stderr: 2 Apr 03:06:19 ntpd[9471]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-02T03:06:19.872 INFO:teuthology.orchestra.run.smithi029.stderr: 2 Apr 03:06:19 ntpd[9471]: can't open /var/log/ntpstats/rawstats.20220402: Permission denied 2022-04-02T03:06:19.872 INFO:teuthology.orchestra.run.smithi029.stderr: 2 Apr 03:06:19 ntpd[9471]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-04-02T03:06:19.872 INFO:teuthology.orchestra.run.smithi029.stderr: 2 Apr 03:06:19 ntpd[9471]: can't open /var/log/ntpstats/peerstats.20220402: Permission denied 2022-04-02T03:06:19.872 INFO:teuthology.orchestra.run.smithi029.stderr: 2 Apr 03:06:19 ntpd[9471]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-02T03:06:19.873 INFO:teuthology.orchestra.run.smithi029.stderr: 2 Apr 03:06:19 ntpd[9471]: can't open /var/log/ntpstats/rawstats.20220402: Permission denied 2022-04-02T03:06:19.873 INFO:teuthology.orchestra.run.smithi029.stderr: 2 Apr 03:06:19 ntpd[9471]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-04-02T03:06:19.873 INFO:teuthology.orchestra.run.smithi029.stderr: 2 Apr 03:06:19 ntpd[9471]: can't open /var/log/ntpstats/peerstats.20220402: Permission denied 2022-04-02T03:06:19.875 INFO:teuthology.orchestra.run.smithi104.stderr: 2 Apr 03:06:19 ntpd[9536]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-02T03:06:19.875 INFO:teuthology.orchestra.run.smithi104.stderr: 2 Apr 03:06:19 ntpd[9536]: can't open /var/log/ntpstats/rawstats.20220402: Permission denied 2022-04-02T03:06:19.875 INFO:teuthology.orchestra.run.smithi104.stderr: 2 Apr 03:06:19 ntpd[9536]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-02T03:06:19.876 INFO:teuthology.orchestra.run.smithi104.stderr: 2 Apr 03:06:19 ntpd[9536]: can't open /var/log/ntpstats/rawstats.20220402: Permission denied 2022-04-02T03:06:19.876 INFO:teuthology.orchestra.run.smithi104.stderr: 2 Apr 03:06:19 ntpd[9536]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-02T03:06:19.877 INFO:teuthology.orchestra.run.smithi104.stderr: 2 Apr 03:06:19 ntpd[9536]: can't open /var/log/ntpstats/rawstats.20220402: Permission denied 2022-04-02T03:06:19.877 INFO:teuthology.orchestra.run.smithi104.stderr: 2 Apr 03:06:19 ntpd[9536]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-04-02T03:06:19.877 INFO:teuthology.orchestra.run.smithi104.stderr: 2 Apr 03:06:19 ntpd[9536]: can't open /var/log/ntpstats/peerstats.20220402: Permission denied 2022-04-02T03:06:19.877 INFO:teuthology.orchestra.run.smithi104.stderr: 2 Apr 03:06:19 ntpd[9536]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-02T03:06:19.878 INFO:teuthology.orchestra.run.smithi104.stderr: 2 Apr 03:06:19 ntpd[9536]: can't open /var/log/ntpstats/rawstats.20220402: Permission denied 2022-04-02T03:06:19.878 INFO:teuthology.orchestra.run.smithi104.stderr: 2 Apr 03:06:19 ntpd[9536]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-04-02T03:06:19.878 INFO:teuthology.orchestra.run.smithi104.stderr: 2 Apr 03:06:19 ntpd[9536]: can't open /var/log/ntpstats/peerstats.20220402: Permission denied 2022-04-02T03:06:19.879 INFO:teuthology.orchestra.run.smithi142.stderr: 2 Apr 03:06:19 ntpd[9599]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-02T03:06:19.879 INFO:teuthology.orchestra.run.smithi142.stderr: 2 Apr 03:06:19 ntpd[9599]: can't open /var/log/ntpstats/rawstats.20220402: Permission denied 2022-04-02T03:06:19.880 INFO:teuthology.orchestra.run.smithi142.stderr: 2 Apr 03:06:19 ntpd[9599]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-02T03:06:19.880 INFO:teuthology.orchestra.run.smithi142.stderr: 2 Apr 03:06:19 ntpd[9599]: can't open /var/log/ntpstats/rawstats.20220402: Permission denied 2022-04-02T03:06:19.880 INFO:teuthology.orchestra.run.smithi142.stderr: 2 Apr 03:06:19 ntpd[9599]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-04-02T03:06:19.880 INFO:teuthology.orchestra.run.smithi142.stderr: 2 Apr 03:06:19 ntpd[9599]: can't open /var/log/ntpstats/peerstats.20220402: Permission denied 2022-04-02T03:06:20.868 INFO:teuthology.orchestra.run.smithi029.stderr: 2 Apr 03:06:20 ntpd[9471]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-02T03:06:20.868 INFO:teuthology.orchestra.run.smithi029.stderr: 2 Apr 03:06:20 ntpd[9471]: can't open /var/log/ntpstats/rawstats.20220402: Permission denied 2022-04-02T03:06:20.868 INFO:teuthology.orchestra.run.smithi029.stderr: 2 Apr 03:06:20 ntpd[9471]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-02T03:06:20.869 INFO:teuthology.orchestra.run.smithi029.stderr: 2 Apr 03:06:20 ntpd[9471]: can't open /var/log/ntpstats/rawstats.20220402: Permission denied 2022-04-02T03:06:20.869 INFO:teuthology.orchestra.run.smithi029.stderr: 2 Apr 03:06:20 ntpd[9471]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-04-02T03:06:20.869 INFO:teuthology.orchestra.run.smithi029.stderr: 2 Apr 03:06:20 ntpd[9471]: can't open /var/log/ntpstats/peerstats.20220402: Permission denied 2022-04-02T03:06:20.870 INFO:teuthology.orchestra.run.smithi104.stderr: 2 Apr 03:06:20 ntpd[9536]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-02T03:06:20.870 INFO:teuthology.orchestra.run.smithi104.stderr: 2 Apr 03:06:20 ntpd[9536]: can't open /var/log/ntpstats/rawstats.20220402: Permission denied 2022-04-02T03:06:20.871 INFO:teuthology.orchestra.run.smithi104.stderr: 2 Apr 03:06:20 ntpd[9536]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-02T03:06:20.871 INFO:teuthology.orchestra.run.smithi104.stderr: 2 Apr 03:06:20 ntpd[9536]: can't open /var/log/ntpstats/rawstats.20220402: Permission denied 2022-04-02T03:06:20.871 INFO:teuthology.orchestra.run.smithi104.stderr: 2 Apr 03:06:20 ntpd[9536]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-04-02T03:06:20.871 INFO:teuthology.orchestra.run.smithi104.stderr: 2 Apr 03:06:20 ntpd[9536]: can't open /var/log/ntpstats/peerstats.20220402: Permission denied 2022-04-02T03:06:20.876 INFO:teuthology.orchestra.run.smithi142.stderr: 2 Apr 03:06:20 ntpd[9599]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-02T03:06:20.876 INFO:teuthology.orchestra.run.smithi142.stderr: 2 Apr 03:06:20 ntpd[9599]: can't open /var/log/ntpstats/rawstats.20220402: Permission denied 2022-04-02T03:06:20.876 INFO:teuthology.orchestra.run.smithi142.stderr: 2 Apr 03:06:20 ntpd[9599]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-02T03:06:20.877 INFO:teuthology.orchestra.run.smithi142.stderr: 2 Apr 03:06:20 ntpd[9599]: can't open /var/log/ntpstats/rawstats.20220402: Permission denied 2022-04-02T03:06:20.877 INFO:teuthology.orchestra.run.smithi142.stderr: 2 Apr 03:06:20 ntpd[9599]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-02T03:06:20.877 INFO:teuthology.orchestra.run.smithi142.stderr: 2 Apr 03:06:20 ntpd[9599]: can't open /var/log/ntpstats/rawstats.20220402: Permission denied 2022-04-02T03:06:20.877 INFO:teuthology.orchestra.run.smithi142.stderr: 2 Apr 03:06:20 ntpd[9599]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-02T03:06:20.878 INFO:teuthology.orchestra.run.smithi142.stderr: 2 Apr 03:06:20 ntpd[9599]: can't open /var/log/ntpstats/rawstats.20220402: Permission denied 2022-04-02T03:06:20.878 INFO:teuthology.orchestra.run.smithi142.stderr: 2 Apr 03:06:20 ntpd[9599]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-04-02T03:06:20.878 INFO:teuthology.orchestra.run.smithi142.stderr: 2 Apr 03:06:20 ntpd[9599]: can't open /var/log/ntpstats/peerstats.20220402: Permission denied 2022-04-02T03:06:20.878 INFO:teuthology.orchestra.run.smithi142.stderr: 2 Apr 03:06:20 ntpd[9599]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-02T03:06:20.879 INFO:teuthology.orchestra.run.smithi142.stderr: 2 Apr 03:06:20 ntpd[9599]: can't open /var/log/ntpstats/rawstats.20220402: Permission denied 2022-04-02T03:06:20.879 INFO:teuthology.orchestra.run.smithi142.stderr: 2 Apr 03:06:20 ntpd[9599]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-04-02T03:06:20.879 INFO:teuthology.orchestra.run.smithi142.stderr: 2 Apr 03:06:20 ntpd[9599]: can't open /var/log/ntpstats/peerstats.20220402: Permission denied 2022-04-02T03:06:20.879 INFO:teuthology.orchestra.run.smithi142.stderr: 2 Apr 03:06:20 ntpd[9599]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-02T03:06:20.879 INFO:teuthology.orchestra.run.smithi142.stderr: 2 Apr 03:06:20 ntpd[9599]: can't open /var/log/ntpstats/rawstats.20220402: Permission denied 2022-04-02T03:06:20.880 INFO:teuthology.orchestra.run.smithi142.stderr: 2 Apr 03:06:20 ntpd[9599]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-04-02T03:06:20.880 INFO:teuthology.orchestra.run.smithi142.stderr: 2 Apr 03:06:20 ntpd[9599]: can't open /var/log/ntpstats/peerstats.20220402: Permission denied 2022-04-02T03:06:21.869 INFO:teuthology.orchestra.run.smithi029.stderr: 2 Apr 03:06:21 ntpd[9471]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-02T03:06:21.870 INFO:teuthology.orchestra.run.smithi029.stderr: 2 Apr 03:06:21 ntpd[9471]: can't open /var/log/ntpstats/rawstats.20220402: Permission denied 2022-04-02T03:06:21.870 INFO:teuthology.orchestra.run.smithi029.stderr: 2 Apr 03:06:21 ntpd[9471]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-02T03:06:21.870 INFO:teuthology.orchestra.run.smithi029.stderr: 2 Apr 03:06:21 ntpd[9471]: can't open /var/log/ntpstats/rawstats.20220402: Permission denied 2022-04-02T03:06:21.871 INFO:teuthology.orchestra.run.smithi029.stderr: 2 Apr 03:06:21 ntpd[9471]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-02T03:06:21.871 INFO:teuthology.orchestra.run.smithi029.stderr: 2 Apr 03:06:21 ntpd[9471]: can't open /var/log/ntpstats/rawstats.20220402: Permission denied 2022-04-02T03:06:21.871 INFO:teuthology.orchestra.run.smithi029.stderr: 2 Apr 03:06:21 ntpd[9471]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-02T03:06:21.871 INFO:teuthology.orchestra.run.smithi029.stderr: 2 Apr 03:06:21 ntpd[9471]: can't open /var/log/ntpstats/rawstats.20220402: Permission denied 2022-04-02T03:06:21.872 INFO:teuthology.orchestra.run.smithi029.stderr: 2 Apr 03:06:21 ntpd[9471]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-04-02T03:06:21.872 INFO:teuthology.orchestra.run.smithi029.stderr: 2 Apr 03:06:21 ntpd[9471]: can't open /var/log/ntpstats/peerstats.20220402: Permission denied 2022-04-02T03:06:21.872 INFO:teuthology.orchestra.run.smithi029.stderr: 2 Apr 03:06:21 ntpd[9471]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-02T03:06:21.872 INFO:teuthology.orchestra.run.smithi029.stderr: 2 Apr 03:06:21 ntpd[9471]: can't open /var/log/ntpstats/rawstats.20220402: Permission denied 2022-04-02T03:06:21.873 INFO:teuthology.orchestra.run.smithi029.stderr: 2 Apr 03:06:21 ntpd[9471]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-04-02T03:06:21.873 INFO:teuthology.orchestra.run.smithi029.stderr: 2 Apr 03:06:21 ntpd[9471]: can't open /var/log/ntpstats/peerstats.20220402: Permission denied 2022-04-02T03:06:21.873 INFO:teuthology.orchestra.run.smithi029.stderr: 2 Apr 03:06:21 ntpd[9471]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-02T03:06:21.873 INFO:teuthology.orchestra.run.smithi029.stderr: 2 Apr 03:06:21 ntpd[9471]: can't open /var/log/ntpstats/rawstats.20220402: Permission denied 2022-04-02T03:06:21.874 INFO:teuthology.orchestra.run.smithi029.stderr: 2 Apr 03:06:21 ntpd[9471]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-04-02T03:06:21.874 INFO:teuthology.orchestra.run.smithi029.stderr: 2 Apr 03:06:21 ntpd[9471]: can't open /var/log/ntpstats/peerstats.20220402: Permission denied 2022-04-02T03:06:21.875 INFO:teuthology.orchestra.run.smithi142.stderr: 2 Apr 03:06:21 ntpd[9599]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-02T03:06:21.876 INFO:teuthology.orchestra.run.smithi142.stderr: 2 Apr 03:06:21 ntpd[9599]: can't open /var/log/ntpstats/rawstats.20220402: Permission denied 2022-04-02T03:06:21.876 INFO:teuthology.orchestra.run.smithi142.stderr: 2 Apr 03:06:21 ntpd[9599]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-02T03:06:21.876 INFO:teuthology.orchestra.run.smithi142.stderr: 2 Apr 03:06:21 ntpd[9599]: can't open /var/log/ntpstats/rawstats.20220402: Permission denied 2022-04-02T03:06:21.876 INFO:teuthology.orchestra.run.smithi142.stderr: 2 Apr 03:06:21 ntpd[9599]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-04-02T03:06:21.877 INFO:teuthology.orchestra.run.smithi142.stderr: 2 Apr 03:06:21 ntpd[9599]: can't open /var/log/ntpstats/peerstats.20220402: Permission denied 2022-04-02T03:06:21.879 INFO:teuthology.orchestra.run.smithi104.stderr: 2 Apr 03:06:21 ntpd[9536]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-02T03:06:21.879 INFO:teuthology.orchestra.run.smithi104.stderr: 2 Apr 03:06:21 ntpd[9536]: can't open /var/log/ntpstats/rawstats.20220402: Permission denied 2022-04-02T03:06:21.879 INFO:teuthology.orchestra.run.smithi104.stderr: 2 Apr 03:06:21 ntpd[9536]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-02T03:06:21.879 INFO:teuthology.orchestra.run.smithi104.stderr: 2 Apr 03:06:21 ntpd[9536]: can't open /var/log/ntpstats/rawstats.20220402: Permission denied 2022-04-02T03:06:21.880 INFO:teuthology.orchestra.run.smithi104.stderr: 2 Apr 03:06:21 ntpd[9536]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-02T03:06:21.880 INFO:teuthology.orchestra.run.smithi104.stderr: 2 Apr 03:06:21 ntpd[9536]: can't open /var/log/ntpstats/rawstats.20220402: Permission denied 2022-04-02T03:06:21.880 INFO:teuthology.orchestra.run.smithi104.stderr: 2 Apr 03:06:21 ntpd[9536]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-02T03:06:21.880 INFO:teuthology.orchestra.run.smithi104.stderr: 2 Apr 03:06:21 ntpd[9536]: can't open /var/log/ntpstats/rawstats.20220402: Permission denied 2022-04-02T03:06:21.881 INFO:teuthology.orchestra.run.smithi104.stderr: 2 Apr 03:06:21 ntpd[9536]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-04-02T03:06:21.881 INFO:teuthology.orchestra.run.smithi104.stderr: 2 Apr 03:06:21 ntpd[9536]: can't open /var/log/ntpstats/peerstats.20220402: Permission denied 2022-04-02T03:06:21.881 INFO:teuthology.orchestra.run.smithi104.stderr: 2 Apr 03:06:21 ntpd[9536]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-02T03:06:21.881 INFO:teuthology.orchestra.run.smithi104.stderr: 2 Apr 03:06:21 ntpd[9536]: can't open /var/log/ntpstats/rawstats.20220402: Permission denied 2022-04-02T03:06:21.882 INFO:teuthology.orchestra.run.smithi104.stderr: 2 Apr 03:06:21 ntpd[9536]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-04-02T03:06:21.882 INFO:teuthology.orchestra.run.smithi104.stderr: 2 Apr 03:06:21 ntpd[9536]: can't open /var/log/ntpstats/peerstats.20220402: Permission denied 2022-04-02T03:06:21.882 INFO:teuthology.orchestra.run.smithi104.stderr: 2 Apr 03:06:21 ntpd[9536]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-02T03:06:21.882 INFO:teuthology.orchestra.run.smithi104.stderr: 2 Apr 03:06:21 ntpd[9536]: can't open /var/log/ntpstats/rawstats.20220402: Permission denied 2022-04-02T03:06:21.883 INFO:teuthology.orchestra.run.smithi104.stderr: 2 Apr 03:06:21 ntpd[9536]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-04-02T03:06:21.883 INFO:teuthology.orchestra.run.smithi104.stderr: 2 Apr 03:06:21 ntpd[9536]: can't open /var/log/ntpstats/peerstats.20220402: Permission denied 2022-04-02T03:06:22.868 INFO:teuthology.orchestra.run.smithi029.stderr: 2 Apr 03:06:22 ntpd[9471]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-02T03:06:22.868 INFO:teuthology.orchestra.run.smithi029.stderr: 2 Apr 03:06:22 ntpd[9471]: can't open /var/log/ntpstats/rawstats.20220402: Permission denied 2022-04-02T03:06:22.869 INFO:teuthology.orchestra.run.smithi029.stderr: 2 Apr 03:06:22 ntpd[9471]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-02T03:06:22.869 INFO:teuthology.orchestra.run.smithi029.stderr: 2 Apr 03:06:22 ntpd[9471]: can't open /var/log/ntpstats/rawstats.20220402: Permission denied 2022-04-02T03:06:22.869 INFO:teuthology.orchestra.run.smithi029.stderr: 2 Apr 03:06:22 ntpd[9471]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-04-02T03:06:22.869 INFO:teuthology.orchestra.run.smithi029.stderr: 2 Apr 03:06:22 ntpd[9471]: can't open /var/log/ntpstats/peerstats.20220402: Permission denied 2022-04-02T03:06:22.870 INFO:teuthology.orchestra.run.smithi104.stderr: 2 Apr 03:06:22 ntpd[9536]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-02T03:06:22.871 INFO:teuthology.orchestra.run.smithi104.stderr: 2 Apr 03:06:22 ntpd[9536]: can't open /var/log/ntpstats/rawstats.20220402: Permission denied 2022-04-02T03:06:22.871 INFO:teuthology.orchestra.run.smithi104.stderr: 2 Apr 03:06:22 ntpd[9536]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-02T03:06:22.871 INFO:teuthology.orchestra.run.smithi104.stderr: 2 Apr 03:06:22 ntpd[9536]: can't open /var/log/ntpstats/rawstats.20220402: Permission denied 2022-04-02T03:06:22.871 INFO:teuthology.orchestra.run.smithi104.stderr: 2 Apr 03:06:22 ntpd[9536]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-04-02T03:06:22.872 INFO:teuthology.orchestra.run.smithi104.stderr: 2 Apr 03:06:22 ntpd[9536]: can't open /var/log/ntpstats/peerstats.20220402: Permission denied 2022-04-02T03:06:22.876 INFO:teuthology.orchestra.run.smithi142.stderr: 2 Apr 03:06:22 ntpd[9599]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-02T03:06:22.876 INFO:teuthology.orchestra.run.smithi142.stderr: 2 Apr 03:06:22 ntpd[9599]: can't open /var/log/ntpstats/rawstats.20220402: Permission denied 2022-04-02T03:06:22.876 INFO:teuthology.orchestra.run.smithi142.stderr: 2 Apr 03:06:22 ntpd[9599]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-02T03:06:22.877 INFO:teuthology.orchestra.run.smithi142.stderr: 2 Apr 03:06:22 ntpd[9599]: can't open /var/log/ntpstats/rawstats.20220402: Permission denied 2022-04-02T03:06:22.877 INFO:teuthology.orchestra.run.smithi142.stderr: 2 Apr 03:06:22 ntpd[9599]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-02T03:06:22.877 INFO:teuthology.orchestra.run.smithi142.stderr: 2 Apr 03:06:22 ntpd[9599]: can't open /var/log/ntpstats/rawstats.20220402: Permission denied 2022-04-02T03:06:22.877 INFO:teuthology.orchestra.run.smithi142.stderr: 2 Apr 03:06:22 ntpd[9599]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-02T03:06:22.878 INFO:teuthology.orchestra.run.smithi142.stderr: 2 Apr 03:06:22 ntpd[9599]: can't open /var/log/ntpstats/rawstats.20220402: Permission denied 2022-04-02T03:06:22.878 INFO:teuthology.orchestra.run.smithi142.stderr: 2 Apr 03:06:22 ntpd[9599]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-04-02T03:06:22.878 INFO:teuthology.orchestra.run.smithi142.stderr: 2 Apr 03:06:22 ntpd[9599]: can't open /var/log/ntpstats/peerstats.20220402: Permission denied 2022-04-02T03:06:22.878 INFO:teuthology.orchestra.run.smithi142.stderr: 2 Apr 03:06:22 ntpd[9599]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-02T03:06:22.879 INFO:teuthology.orchestra.run.smithi142.stderr: 2 Apr 03:06:22 ntpd[9599]: can't open /var/log/ntpstats/rawstats.20220402: Permission denied 2022-04-02T03:06:22.879 INFO:teuthology.orchestra.run.smithi142.stderr: 2 Apr 03:06:22 ntpd[9599]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-04-02T03:06:22.879 INFO:teuthology.orchestra.run.smithi142.stderr: 2 Apr 03:06:22 ntpd[9599]: can't open /var/log/ntpstats/peerstats.20220402: Permission denied 2022-04-02T03:06:22.879 INFO:teuthology.orchestra.run.smithi142.stderr: 2 Apr 03:06:22 ntpd[9599]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-02T03:06:22.879 INFO:teuthology.orchestra.run.smithi142.stderr: 2 Apr 03:06:22 ntpd[9599]: can't open /var/log/ntpstats/rawstats.20220402: Permission denied 2022-04-02T03:06:22.880 INFO:teuthology.orchestra.run.smithi142.stderr: 2 Apr 03:06:22 ntpd[9599]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-04-02T03:06:22.880 INFO:teuthology.orchestra.run.smithi142.stderr: 2 Apr 03:06:22 ntpd[9599]: can't open /var/log/ntpstats/peerstats.20220402: Permission denied 2022-04-02T03:06:23.869 INFO:teuthology.orchestra.run.smithi029.stderr: 2 Apr 03:06:23 ntpd[9471]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-02T03:06:23.870 INFO:teuthology.orchestra.run.smithi029.stderr: 2 Apr 03:06:23 ntpd[9471]: can't open /var/log/ntpstats/rawstats.20220402: Permission denied 2022-04-02T03:06:23.870 INFO:teuthology.orchestra.run.smithi029.stderr: 2 Apr 03:06:23 ntpd[9471]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-02T03:06:23.870 INFO:teuthology.orchestra.run.smithi029.stderr: 2 Apr 03:06:23 ntpd[9471]: can't open /var/log/ntpstats/rawstats.20220402: Permission denied 2022-04-02T03:06:23.870 INFO:teuthology.orchestra.run.smithi029.stderr: 2 Apr 03:06:23 ntpd[9471]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-02T03:06:23.871 INFO:teuthology.orchestra.run.smithi029.stderr: 2 Apr 03:06:23 ntpd[9471]: can't open /var/log/ntpstats/rawstats.20220402: Permission denied 2022-04-02T03:06:23.871 INFO:teuthology.orchestra.run.smithi029.stderr: 2 Apr 03:06:23 ntpd[9471]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-02T03:06:23.871 INFO:teuthology.orchestra.run.smithi029.stderr: 2 Apr 03:06:23 ntpd[9471]: can't open /var/log/ntpstats/rawstats.20220402: Permission denied 2022-04-02T03:06:23.871 INFO:teuthology.orchestra.run.smithi029.stderr: 2 Apr 03:06:23 ntpd[9471]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-04-02T03:06:23.872 INFO:teuthology.orchestra.run.smithi029.stderr: 2 Apr 03:06:23 ntpd[9471]: can't open /var/log/ntpstats/peerstats.20220402: Permission denied 2022-04-02T03:06:23.872 INFO:teuthology.orchestra.run.smithi029.stderr: 2 Apr 03:06:23 ntpd[9471]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-02T03:06:23.872 INFO:teuthology.orchestra.run.smithi029.stderr: 2 Apr 03:06:23 ntpd[9471]: can't open /var/log/ntpstats/rawstats.20220402: Permission denied 2022-04-02T03:06:23.872 INFO:teuthology.orchestra.run.smithi029.stderr: 2 Apr 03:06:23 ntpd[9471]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-04-02T03:06:23.872 INFO:teuthology.orchestra.run.smithi029.stderr: 2 Apr 03:06:23 ntpd[9471]: can't open /var/log/ntpstats/peerstats.20220402: Permission denied 2022-04-02T03:06:23.873 INFO:teuthology.orchestra.run.smithi029.stdout: 2 Apr 03:06:23 ntpd[9471]: ntpd: time slew -0.000806 s 2022-04-02T03:06:23.873 INFO:teuthology.orchestra.run.smithi029.stdout:ntpd: time slew -0.000806s 2022-04-02T03:06:23.874 INFO:teuthology.orchestra.run.smithi029.stderr: 2 Apr 03:06:23 ntpd[9471]: can't open /var/log/ntpstats/loopstats.20220402: Permission denied 2022-04-02T03:06:23.876 INFO:teuthology.orchestra.run.smithi142.stderr: 2 Apr 03:06:23 ntpd[9599]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-02T03:06:23.876 INFO:teuthology.orchestra.run.smithi142.stderr: 2 Apr 03:06:23 ntpd[9599]: can't open /var/log/ntpstats/rawstats.20220402: Permission denied 2022-04-02T03:06:23.876 INFO:teuthology.orchestra.run.smithi142.stderr: 2 Apr 03:06:23 ntpd[9599]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-02T03:06:23.876 INFO:teuthology.orchestra.run.smithi142.stderr: 2 Apr 03:06:23 ntpd[9599]: can't open /var/log/ntpstats/rawstats.20220402: Permission denied 2022-04-02T03:06:23.877 INFO:teuthology.orchestra.run.smithi142.stderr: 2 Apr 03:06:23 ntpd[9599]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-04-02T03:06:23.877 INFO:teuthology.orchestra.run.smithi142.stderr: 2 Apr 03:06:23 ntpd[9599]: can't open /var/log/ntpstats/peerstats.20220402: Permission denied 2022-04-02T03:06:23.878 INFO:teuthology.orchestra.run.smithi142.stdout: 2 Apr 03:06:23 ntpd[9599]: ntpd: time slew +0.000114 s 2022-04-02T03:06:23.879 INFO:teuthology.orchestra.run.smithi142.stdout:ntpd: time slew +0.000114s 2022-04-02T03:06:23.879 INFO:teuthology.orchestra.run.smithi142.stderr: 2 Apr 03:06:23 ntpd[9599]: can't open /var/log/ntpstats/loopstats.20220402: Permission denied 2022-04-02T03:06:23.880 INFO:teuthology.orchestra.run.smithi104.stderr: 2 Apr 03:06:23 ntpd[9536]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-02T03:06:23.880 INFO:teuthology.orchestra.run.smithi104.stderr: 2 Apr 03:06:23 ntpd[9536]: can't open /var/log/ntpstats/rawstats.20220402: Permission denied 2022-04-02T03:06:23.881 INFO:teuthology.orchestra.run.smithi104.stderr: 2 Apr 03:06:23 ntpd[9536]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-02T03:06:23.881 INFO:teuthology.orchestra.run.smithi104.stderr: 2 Apr 03:06:23 ntpd[9536]: can't open /var/log/ntpstats/rawstats.20220402: Permission denied 2022-04-02T03:06:23.881 INFO:teuthology.orchestra.run.smithi104.stderr: 2 Apr 03:06:23 ntpd[9536]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-02T03:06:23.881 INFO:teuthology.orchestra.run.smithi104.stderr: 2 Apr 03:06:23 ntpd[9536]: can't open /var/log/ntpstats/rawstats.20220402: Permission denied 2022-04-02T03:06:23.882 INFO:teuthology.orchestra.run.smithi104.stderr: 2 Apr 03:06:23 ntpd[9536]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-02T03:06:23.882 INFO:teuthology.orchestra.run.smithi104.stderr: 2 Apr 03:06:23 ntpd[9536]: can't open /var/log/ntpstats/rawstats.20220402: Permission denied 2022-04-02T03:06:23.882 INFO:teuthology.orchestra.run.smithi104.stderr: 2 Apr 03:06:23 ntpd[9536]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-04-02T03:06:23.882 INFO:teuthology.orchestra.run.smithi104.stderr: 2 Apr 03:06:23 ntpd[9536]: can't open /var/log/ntpstats/peerstats.20220402: Permission denied 2022-04-02T03:06:23.883 INFO:teuthology.orchestra.run.smithi104.stderr: 2 Apr 03:06:23 ntpd[9536]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-02T03:06:23.883 INFO:teuthology.orchestra.run.smithi104.stderr: 2 Apr 03:06:23 ntpd[9536]: can't open /var/log/ntpstats/rawstats.20220402: Permission denied 2022-04-02T03:06:23.883 INFO:teuthology.orchestra.run.smithi104.stderr: 2 Apr 03:06:23 ntpd[9536]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-04-02T03:06:23.883 INFO:teuthology.orchestra.run.smithi104.stderr: 2 Apr 03:06:23 ntpd[9536]: can't open /var/log/ntpstats/peerstats.20220402: Permission denied 2022-04-02T03:06:23.884 INFO:teuthology.orchestra.run.smithi104.stdout: 2 Apr 03:06:23 ntpd[9536]: ntpd: time slew +0.000408 s 2022-04-02T03:06:23.884 INFO:teuthology.orchestra.run.smithi104.stdout:ntpd: time slew +0.000408s 2022-04-02T03:06:23.884 INFO:teuthology.orchestra.run.smithi104.stderr: 2 Apr 03:06:23 ntpd[9536]: can't open /var/log/ntpstats/loopstats.20220402: Permission denied 2022-04-02T03:06:23.929 INFO:teuthology.orchestra.run.smithi029.stdout: remote refid st t when poll reach delay offset jitter 2022-04-02T03:06:23.930 INFO:teuthology.orchestra.run.smithi029.stdout:============================================================================== 2022-04-02T03:06:23.930 INFO:teuthology.orchestra.run.smithi029.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-04-02T03:06:23.930 INFO:teuthology.orchestra.run.smithi029.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-04-02T03:06:23.930 INFO:teuthology.orchestra.run.smithi029.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-04-02T03:06:23.931 INFO:teuthology.orchestra.run.smithi029.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-04-02T03:06:23.984 INFO:teuthology.orchestra.run.smithi104.stdout: remote refid st t when poll reach delay offset jitter 2022-04-02T03:06:23.984 INFO:teuthology.orchestra.run.smithi104.stdout:============================================================================== 2022-04-02T03:06:23.984 INFO:teuthology.orchestra.run.smithi104.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-04-02T03:06:23.985 INFO:teuthology.orchestra.run.smithi104.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-04-02T03:06:23.985 INFO:teuthology.orchestra.run.smithi104.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-04-02T03:06:23.985 INFO:teuthology.orchestra.run.smithi104.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-04-02T03:06:24.037 INFO:teuthology.orchestra.run.smithi142.stdout: remote refid st t when poll reach delay offset jitter 2022-04-02T03:06:24.038 INFO:teuthology.orchestra.run.smithi142.stdout:============================================================================== 2022-04-02T03:06:24.038 INFO:teuthology.orchestra.run.smithi142.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-04-02T03:06:24.039 INFO:teuthology.orchestra.run.smithi142.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-04-02T03:06:24.039 INFO:teuthology.orchestra.run.smithi142.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-04-02T03:06:24.039 INFO:teuthology.orchestra.run.smithi142.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-04-02T03:06:24.040 INFO:teuthology.run_tasks:Running task kubeadm... 2022-04-02T03:06:24.057 INFO:tasks.kubeadm:Kubeadm start 2022-04-02T03:06:24.058 INFO:tasks.kubeadm:Config: {'pod_network': 'flannel', 'version': '1.21'} 2022-04-02T03:06:24.059 DEBUG:teuthology.orchestra.run.smithi029:> sudo modprobe br_netfilter && sudo sysctl net.bridge.bridge-nf-call-ip6tables=1 && sudo sysctl net.bridge.bridge-nf-call-iptables=1 && sudo sysctl net.ipv4.ip_forward=1 && sudo swapoff -a 2022-04-02T03:06:24.063 DEBUG:teuthology.orchestra.run.smithi104:> sudo modprobe br_netfilter && sudo sysctl net.bridge.bridge-nf-call-ip6tables=1 && sudo sysctl net.bridge.bridge-nf-call-iptables=1 && sudo sysctl net.ipv4.ip_forward=1 && sudo swapoff -a 2022-04-02T03:06:24.065 DEBUG:teuthology.orchestra.run.smithi142:> sudo modprobe br_netfilter && sudo sysctl net.bridge.bridge-nf-call-ip6tables=1 && sudo sysctl net.bridge.bridge-nf-call-iptables=1 && sudo sysctl net.ipv4.ip_forward=1 && sudo swapoff -a 2022-04-02T03:06:24.083 INFO:teuthology.orchestra.run.smithi029.stdout:net.bridge.bridge-nf-call-ip6tables = 1 2022-04-02T03:06:24.090 INFO:teuthology.orchestra.run.smithi104.stdout:net.bridge.bridge-nf-call-ip6tables = 1 2022-04-02T03:06:24.091 INFO:teuthology.orchestra.run.smithi029.stdout:net.bridge.bridge-nf-call-iptables = 1 2022-04-02T03:06:24.098 INFO:teuthology.orchestra.run.smithi029.stdout:net.ipv4.ip_forward = 1 2022-04-02T03:06:24.100 INFO:teuthology.orchestra.run.smithi104.stdout:net.bridge.bridge-nf-call-iptables = 1 2022-04-02T03:06:24.108 INFO:teuthology.orchestra.run.smithi104.stdout:net.ipv4.ip_forward = 1 2022-04-02T03:06:24.108 INFO:teuthology.orchestra.run.smithi142.stdout:net.bridge.bridge-nf-call-ip6tables = 1 2022-04-02T03:06:24.116 INFO:teuthology.orchestra.run.smithi142.stdout:net.bridge.bridge-nf-call-iptables = 1 2022-04-02T03:06:24.124 INFO:teuthology.orchestra.run.smithi142.stdout:net.ipv4.ip_forward = 1 2022-04-02T03:06:24.184 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2022-04-02T03:06:24.185 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/etc/docker/daemon.json of=/dev/stdout 2022-04-02T03:06:24.202 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2022-04-02T03:06:24.202 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/etc/docker/daemon.json 2022-04-02T03:06:24.217 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2022-04-02T03:06:24.218 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd if=/etc/docker/daemon.json of=/dev/stdout 2022-04-02T03:06:24.235 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2022-04-02T03:06:24.235 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/etc/docker/daemon.json 2022-04-02T03:06:24.246 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2022-04-02T03:06:24.247 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd if=/etc/docker/daemon.json of=/dev/stdout 2022-04-02T03:06:24.263 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2022-04-02T03:06:24.264 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd of=/etc/docker/daemon.json 2022-04-02T03:06:24.278 DEBUG:teuthology.orchestra.run.smithi029:> sudo systemctl restart docker || true 2022-04-02T03:06:24.281 DEBUG:teuthology.orchestra.run.smithi104:> sudo systemctl restart docker || true 2022-04-02T03:06:24.289 DEBUG:teuthology.orchestra.run.smithi142:> sudo systemctl restart docker || true 2022-04-02T03:06:25.425 INFO:tasks.kubeadm:Identifying pod subnet 2022-04-02T03:06:25.437 INFO:tasks.kubeadm:172.21.15.29 is in 172.21.0.0/20 at pos 3868 2022-04-02T03:06:25.462 INFO:tasks.kubeadm:Pod subnet: 10.248.224.0/21 2022-04-02T03:06:25.463 INFO:tasks.kubeadm:Installing kube{adm,ctl,let} 2022-04-02T03:06:25.464 DEBUG:teuthology.orchestra.run.smithi029:> sudo apt update && sudo apt install -y apt-transport-https ca-certificates curl && sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg && echo 'deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main' | sudo tee /etc/apt/sources.list.d/kubernetes.list && sudo apt update && sudo apt install -y kubelet kubeadm kubectl bridge-utils 2022-04-02T03:06:25.468 DEBUG:teuthology.orchestra.run.smithi104:> sudo apt update && sudo apt install -y apt-transport-https ca-certificates curl && sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg && echo 'deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main' | sudo tee /etc/apt/sources.list.d/kubernetes.list && sudo apt update && sudo apt install -y kubelet kubeadm kubectl bridge-utils 2022-04-02T03:06:25.471 DEBUG:teuthology.orchestra.run.smithi142:> sudo apt update && sudo apt install -y apt-transport-https ca-certificates curl && sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg && echo 'deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main' | sudo tee /etc/apt/sources.list.d/kubernetes.list && sudo apt update && sudo apt install -y kubelet kubeadm kubectl bridge-utils 2022-04-02T03:06:25.491 INFO:teuthology.orchestra.run.smithi104.stderr: 2022-04-02T03:06:25.491 INFO:teuthology.orchestra.run.smithi104.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-04-02T03:06:25.492 INFO:teuthology.orchestra.run.smithi104.stderr: 2022-04-02T03:06:25.494 INFO:teuthology.orchestra.run.smithi029.stderr: 2022-04-02T03:06:25.495 INFO:teuthology.orchestra.run.smithi029.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-04-02T03:06:25.495 INFO:teuthology.orchestra.run.smithi029.stderr: 2022-04-02T03:06:25.496 INFO:teuthology.orchestra.run.smithi142.stderr: 2022-04-02T03:06:25.496 INFO:teuthology.orchestra.run.smithi142.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-04-02T03:06:25.496 INFO:teuthology.orchestra.run.smithi142.stderr: 2022-04-02T03:06:25.608 INFO:teuthology.orchestra.run.smithi104.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-04-02T03:06:25.609 INFO:teuthology.orchestra.run.smithi104.stdout:Hit:2 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-04-02T03:06:25.609 INFO:teuthology.orchestra.run.smithi142.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-04-02T03:06:25.615 INFO:teuthology.orchestra.run.smithi029.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-04-02T03:06:25.615 INFO:teuthology.orchestra.run.smithi104.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-04-02T03:06:25.619 INFO:teuthology.orchestra.run.smithi142.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-04-02T03:06:25.636 INFO:teuthology.orchestra.run.smithi104.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-04-02T03:06:25.639 INFO:teuthology.orchestra.run.smithi142.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-04-02T03:06:25.720 INFO:teuthology.orchestra.run.smithi142.stdout:Hit:4 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-04-02T03:06:25.722 INFO:teuthology.orchestra.run.smithi029.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2022-04-02T03:06:25.802 INFO:teuthology.orchestra.run.smithi029.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-04-02T03:06:25.882 INFO:teuthology.orchestra.run.smithi029.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-04-02T03:06:27.086 INFO:teuthology.orchestra.run.smithi104.stdout:Reading package lists... 2022-04-02T03:06:27.122 INFO:teuthology.orchestra.run.smithi142.stdout:Reading package lists... 2022-04-02T03:06:27.150 INFO:teuthology.orchestra.run.smithi029.stdout:Reading package lists... 2022-04-02T03:06:27.257 INFO:teuthology.orchestra.run.smithi104.stdout:Building dependency tree... 2022-04-02T03:06:27.257 INFO:teuthology.orchestra.run.smithi104.stdout:Reading state information... 2022-04-02T03:06:27.300 INFO:teuthology.orchestra.run.smithi142.stdout:Building dependency tree... 2022-04-02T03:06:27.301 INFO:teuthology.orchestra.run.smithi142.stdout:Reading state information... 2022-04-02T03:06:27.328 INFO:teuthology.orchestra.run.smithi029.stdout:Building dependency tree... 2022-04-02T03:06:27.329 INFO:teuthology.orchestra.run.smithi029.stdout:Reading state information... 2022-04-02T03:06:27.349 INFO:teuthology.orchestra.run.smithi104.stdout:109 packages can be upgraded. Run 'apt list --upgradable' to see them. 2022-04-02T03:06:27.364 INFO:teuthology.orchestra.run.smithi104.stderr: 2022-04-02T03:06:27.365 INFO:teuthology.orchestra.run.smithi104.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-04-02T03:06:27.365 INFO:teuthology.orchestra.run.smithi104.stderr: 2022-04-02T03:06:27.390 INFO:teuthology.orchestra.run.smithi142.stdout:109 packages can be upgraded. Run 'apt list --upgradable' to see them. 2022-04-02T03:06:27.407 INFO:teuthology.orchestra.run.smithi142.stderr: 2022-04-02T03:06:27.407 INFO:teuthology.orchestra.run.smithi142.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-04-02T03:06:27.407 INFO:teuthology.orchestra.run.smithi142.stderr: 2022-04-02T03:06:27.420 INFO:teuthology.orchestra.run.smithi104.stdout:Reading package lists... 2022-04-02T03:06:27.423 INFO:teuthology.orchestra.run.smithi029.stdout:109 packages can be upgraded. Run 'apt list --upgradable' to see them. 2022-04-02T03:06:27.439 INFO:teuthology.orchestra.run.smithi029.stderr: 2022-04-02T03:06:27.440 INFO:teuthology.orchestra.run.smithi029.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-04-02T03:06:27.440 INFO:teuthology.orchestra.run.smithi029.stderr: 2022-04-02T03:06:27.460 INFO:teuthology.orchestra.run.smithi142.stdout:Reading package lists... 2022-04-02T03:06:27.498 INFO:teuthology.orchestra.run.smithi029.stdout:Reading package lists... 2022-04-02T03:06:27.597 INFO:teuthology.orchestra.run.smithi104.stdout:Building dependency tree... 2022-04-02T03:06:27.598 INFO:teuthology.orchestra.run.smithi104.stdout:Reading state information... 2022-04-02T03:06:27.636 INFO:teuthology.orchestra.run.smithi142.stdout:Building dependency tree... 2022-04-02T03:06:27.637 INFO:teuthology.orchestra.run.smithi142.stdout:Reading state information... 2022-04-02T03:06:27.679 INFO:teuthology.orchestra.run.smithi029.stdout:Building dependency tree... 2022-04-02T03:06:27.680 INFO:teuthology.orchestra.run.smithi029.stdout:Reading state information... 2022-04-02T03:06:27.786 INFO:teuthology.orchestra.run.smithi104.stdout:ca-certificates is already the newest version (20210119~20.04.2). 2022-04-02T03:06:27.787 INFO:teuthology.orchestra.run.smithi104.stdout:curl is already the newest version (7.68.0-1ubuntu2.7). 2022-04-02T03:06:27.787 INFO:teuthology.orchestra.run.smithi104.stdout:The following NEW packages will be installed: 2022-04-02T03:06:27.788 INFO:teuthology.orchestra.run.smithi104.stdout: apt-transport-https 2022-04-02T03:06:27.815 INFO:teuthology.orchestra.run.smithi142.stdout:ca-certificates is already the newest version (20210119~20.04.2). 2022-04-02T03:06:27.816 INFO:teuthology.orchestra.run.smithi142.stdout:curl is already the newest version (7.68.0-1ubuntu2.7). 2022-04-02T03:06:27.816 INFO:teuthology.orchestra.run.smithi142.stdout:The following NEW packages will be installed: 2022-04-02T03:06:27.816 INFO:teuthology.orchestra.run.smithi142.stdout: apt-transport-https 2022-04-02T03:06:27.848 INFO:teuthology.orchestra.run.smithi104.stdout:0 upgraded, 1 newly installed, 0 to remove and 109 not upgraded. 2022-04-02T03:06:27.849 INFO:teuthology.orchestra.run.smithi104.stdout:Need to get 4,680 B of archives. 2022-04-02T03:06:27.849 INFO:teuthology.orchestra.run.smithi104.stdout:After this operation, 162 kB of additional disk space will be used. 2022-04-02T03:06:27.849 INFO:teuthology.orchestra.run.smithi104.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 apt-transport-https all 2.0.6 [4,680 B] 2022-04-02T03:06:27.876 INFO:teuthology.orchestra.run.smithi029.stdout:ca-certificates is already the newest version (20210119~20.04.2). 2022-04-02T03:06:27.877 INFO:teuthology.orchestra.run.smithi029.stdout:curl is already the newest version (7.68.0-1ubuntu2.7). 2022-04-02T03:06:27.877 INFO:teuthology.orchestra.run.smithi029.stdout:The following NEW packages will be installed: 2022-04-02T03:06:27.878 INFO:teuthology.orchestra.run.smithi029.stdout: apt-transport-https 2022-04-02T03:06:27.941 INFO:teuthology.orchestra.run.smithi029.stdout:0 upgraded, 1 newly installed, 0 to remove and 109 not upgraded. 2022-04-02T03:06:27.941 INFO:teuthology.orchestra.run.smithi029.stdout:Need to get 4,680 B of archives. 2022-04-02T03:06:27.941 INFO:teuthology.orchestra.run.smithi029.stdout:After this operation, 162 kB of additional disk space will be used. 2022-04-02T03:06:27.942 INFO:teuthology.orchestra.run.smithi029.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 apt-transport-https all 2.0.6 [4,680 B] 2022-04-02T03:06:27.996 INFO:teuthology.orchestra.run.smithi142.stdout:0 upgraded, 1 newly installed, 0 to remove and 109 not upgraded. 2022-04-02T03:06:27.997 INFO:teuthology.orchestra.run.smithi142.stdout:Need to get 4,680 B of archives. 2022-04-02T03:06:27.997 INFO:teuthology.orchestra.run.smithi142.stdout:After this operation, 162 kB of additional disk space will be used. 2022-04-02T03:06:27.997 INFO:teuthology.orchestra.run.smithi142.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 apt-transport-https all 2.0.6 [4,680 B] 2022-04-02T03:06:28.086 INFO:teuthology.orchestra.run.smithi104.stderr:debconf: unable to initialize frontend: Dialog 2022-04-02T03:06:28.086 INFO:teuthology.orchestra.run.smithi104.stderr:debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) 2022-04-02T03:06:28.087 INFO:teuthology.orchestra.run.smithi104.stderr:debconf: falling back to frontend: Readline 2022-04-02T03:06:28.108 INFO:teuthology.orchestra.run.smithi104.stderr:debconf: unable to initialize frontend: Readline 2022-04-02T03:06:28.108 INFO:teuthology.orchestra.run.smithi104.stderr:debconf: (This frontend requires a controlling tty.) 2022-04-02T03:06:28.108 INFO:teuthology.orchestra.run.smithi104.stderr:debconf: falling back to frontend: Teletype 2022-04-02T03:06:28.114 INFO:teuthology.orchestra.run.smithi104.stderr:dpkg-preconfigure: unable to re-open stdin: 2022-04-02T03:06:28.165 INFO:teuthology.orchestra.run.smithi104.stdout:Fetched 4,680 B in 0s (88.0 kB/s) 2022-04-02T03:06:28.180 INFO:teuthology.orchestra.run.smithi029.stderr:debconf: unable to initialize frontend: Dialog 2022-04-02T03:06:28.180 INFO:teuthology.orchestra.run.smithi029.stderr:debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) 2022-04-02T03:06:28.180 INFO:teuthology.orchestra.run.smithi029.stderr:debconf: falling back to frontend: Readline 2022-04-02T03:06:28.199 INFO:teuthology.orchestra.run.smithi029.stderr:debconf: unable to initialize frontend: Readline 2022-04-02T03:06:28.200 INFO:teuthology.orchestra.run.smithi029.stderr:debconf: (This frontend requires a controlling tty.) 2022-04-02T03:06:28.200 INFO:teuthology.orchestra.run.smithi029.stderr:debconf: falling back to frontend: Teletype 2022-04-02T03:06:28.206 INFO:teuthology.orchestra.run.smithi029.stderr:dpkg-preconfigure: unable to re-open stdin: 2022-04-02T03:06:28.231 INFO:teuthology.orchestra.run.smithi142.stderr:debconf: unable to initialize frontend: Dialog 2022-04-02T03:06:28.231 INFO:teuthology.orchestra.run.smithi142.stderr:debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) 2022-04-02T03:06:28.232 INFO:teuthology.orchestra.run.smithi142.stderr:debconf: falling back to frontend: Readline 2022-04-02T03:06:28.242 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package apt-transport-https. 2022-04-02T03:06:28.247 INFO:teuthology.orchestra.run.smithi142.stderr:debconf: unable to initialize frontend: Readline 2022-04-02T03:06:28.247 INFO:teuthology.orchestra.run.smithi142.stderr:debconf: (This frontend requires a controlling tty.) 2022-04-02T03:06:28.247 INFO:teuthology.orchestra.run.smithi142.stderr:debconf: falling back to frontend: Teletype 2022-04-02T03:06:28.252 INFO:teuthology.orchestra.run.smithi142.stderr:dpkg-preconfigure: unable to re-open stdin: 2022-04-02T03:06:28.261 INFO:teuthology.orchestra.run.smithi029.stdout:Fetched 4,680 B in 0s (88.8 kB/s) 2022-04-02T03:06:28.287 INFO:teuthology.orchestra.run.smithi104.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109240 files and directories currently installed.) 2022-04-02T03:06:28.290 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../apt-transport-https_2.0.6_all.deb ... 2022-04-02T03:06:28.302 INFO:teuthology.orchestra.run.smithi142.stdout:Fetched 4,680 B in 0s (27.1 kB/s) 2022-04-02T03:06:28.324 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking apt-transport-https (2.0.6) ... 2022-04-02T03:06:28.337 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package apt-transport-https. 2022-04-02T03:06:28.379 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package apt-transport-https. 2022-04-02T03:06:28.381 INFO:teuthology.orchestra.run.smithi029.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109240 files and directories currently installed.) 2022-04-02T03:06:28.384 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../apt-transport-https_2.0.6_all.deb ... 2022-04-02T03:06:28.422 INFO:teuthology.orchestra.run.smithi142.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109240 files and directories currently installed.) 2022-04-02T03:06:28.425 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../apt-transport-https_2.0.6_all.deb ... 2022-04-02T03:06:28.427 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking apt-transport-https (2.0.6) ... 2022-04-02T03:06:28.470 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking apt-transport-https (2.0.6) ... 2022-04-02T03:06:28.594 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up apt-transport-https (2.0.6) ... 2022-04-02T03:06:28.709 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up apt-transport-https (2.0.6) ... 2022-04-02T03:06:28.735 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up apt-transport-https (2.0.6) ... 2022-04-02T03:06:29.134 INFO:teuthology.orchestra.run.smithi104.stdout:deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main 2022-04-02T03:06:29.145 INFO:teuthology.orchestra.run.smithi104.stderr: 2022-04-02T03:06:29.145 INFO:teuthology.orchestra.run.smithi104.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-04-02T03:06:29.145 INFO:teuthology.orchestra.run.smithi104.stderr: 2022-04-02T03:06:29.252 INFO:teuthology.orchestra.run.smithi029.stdout:deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main 2022-04-02T03:06:29.266 INFO:teuthology.orchestra.run.smithi029.stderr: 2022-04-02T03:06:29.267 INFO:teuthology.orchestra.run.smithi029.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-04-02T03:06:29.267 INFO:teuthology.orchestra.run.smithi029.stderr: 2022-04-02T03:06:29.268 INFO:teuthology.orchestra.run.smithi142.stdout:deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main 2022-04-02T03:06:29.275 INFO:teuthology.orchestra.run.smithi104.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-04-02T03:06:29.276 INFO:teuthology.orchestra.run.smithi104.stdout:Hit:2 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-04-02T03:06:29.279 INFO:teuthology.orchestra.run.smithi142.stderr: 2022-04-02T03:06:29.280 INFO:teuthology.orchestra.run.smithi142.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-04-02T03:06:29.280 INFO:teuthology.orchestra.run.smithi142.stderr: 2022-04-02T03:06:29.284 INFO:teuthology.orchestra.run.smithi104.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-04-02T03:06:29.304 INFO:teuthology.orchestra.run.smithi104.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-04-02T03:06:29.396 INFO:teuthology.orchestra.run.smithi029.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-04-02T03:06:29.405 INFO:teuthology.orchestra.run.smithi029.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-04-02T03:06:29.410 INFO:teuthology.orchestra.run.smithi142.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-04-02T03:06:29.424 INFO:teuthology.orchestra.run.smithi029.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-04-02T03:06:29.505 INFO:teuthology.orchestra.run.smithi029.stdout:Hit:4 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-04-02T03:06:29.520 INFO:teuthology.orchestra.run.smithi142.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2022-04-02T03:06:29.600 INFO:teuthology.orchestra.run.smithi142.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-04-02T03:06:29.681 INFO:teuthology.orchestra.run.smithi142.stdout:Hit:5 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-04-02T03:06:29.922 INFO:teuthology.orchestra.run.smithi029.stdout:Get:5 https://packages.cloud.google.com/apt kubernetes-xenial InRelease [9,383 B] 2022-04-02T03:06:30.060 INFO:teuthology.orchestra.run.smithi029.stdout:Get:6 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 Packages [54.7 kB] 2022-04-02T03:06:30.097 INFO:teuthology.orchestra.run.smithi142.stdout:Get:4 https://packages.cloud.google.com/apt kubernetes-xenial InRelease [9,383 B] 2022-04-02T03:06:30.165 INFO:teuthology.orchestra.run.smithi104.stdout:Get:5 https://packages.cloud.google.com/apt kubernetes-xenial InRelease [9,383 B] 2022-04-02T03:06:30.207 INFO:teuthology.orchestra.run.smithi029.stdout:Fetched 64.1 kB in 1s (81.3 kB/s) 2022-04-02T03:06:30.218 INFO:teuthology.orchestra.run.smithi142.stdout:Get:6 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 Packages [54.7 kB] 2022-04-02T03:06:30.260 INFO:teuthology.orchestra.run.smithi104.stdout:Get:6 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 Packages [54.7 kB] 2022-04-02T03:06:30.357 INFO:teuthology.orchestra.run.smithi142.stdout:Fetched 64.1 kB in 1s (68.6 kB/s) 2022-04-02T03:06:30.388 INFO:teuthology.orchestra.run.smithi104.stdout:Fetched 64.1 kB in 1s (58.4 kB/s) 2022-04-02T03:06:31.110 INFO:teuthology.orchestra.run.smithi029.stdout:Reading package lists... 2022-04-02T03:06:31.219 INFO:teuthology.orchestra.run.smithi142.stdout:Reading package lists... 2022-04-02T03:06:31.238 INFO:teuthology.orchestra.run.smithi104.stdout:Reading package lists... 2022-04-02T03:06:31.295 INFO:teuthology.orchestra.run.smithi029.stdout:Building dependency tree... 2022-04-02T03:06:31.296 INFO:teuthology.orchestra.run.smithi029.stdout:Reading state information... 2022-04-02T03:06:31.393 INFO:teuthology.orchestra.run.smithi029.stdout:109 packages can be upgraded. Run 'apt list --upgradable' to see them. 2022-04-02T03:06:31.395 INFO:teuthology.orchestra.run.smithi142.stdout:Building dependency tree... 2022-04-02T03:06:31.395 INFO:teuthology.orchestra.run.smithi142.stdout:Reading state information... 2022-04-02T03:06:31.411 INFO:teuthology.orchestra.run.smithi029.stderr: 2022-04-02T03:06:31.412 INFO:teuthology.orchestra.run.smithi029.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-04-02T03:06:31.412 INFO:teuthology.orchestra.run.smithi029.stderr: 2022-04-02T03:06:31.418 INFO:teuthology.orchestra.run.smithi104.stdout:Building dependency tree... 2022-04-02T03:06:31.418 INFO:teuthology.orchestra.run.smithi104.stdout:Reading state information... 2022-04-02T03:06:31.466 INFO:teuthology.orchestra.run.smithi029.stdout:Reading package lists... 2022-04-02T03:06:31.483 INFO:teuthology.orchestra.run.smithi142.stdout:109 packages can be upgraded. Run 'apt list --upgradable' to see them. 2022-04-02T03:06:31.495 INFO:teuthology.orchestra.run.smithi142.stderr: 2022-04-02T03:06:31.495 INFO:teuthology.orchestra.run.smithi142.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-04-02T03:06:31.495 INFO:teuthology.orchestra.run.smithi142.stderr: 2022-04-02T03:06:31.508 INFO:teuthology.orchestra.run.smithi104.stdout:109 packages can be upgraded. Run 'apt list --upgradable' to see them. 2022-04-02T03:06:31.524 INFO:teuthology.orchestra.run.smithi104.stderr: 2022-04-02T03:06:31.524 INFO:teuthology.orchestra.run.smithi104.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-04-02T03:06:31.525 INFO:teuthology.orchestra.run.smithi104.stderr: 2022-04-02T03:06:31.552 INFO:teuthology.orchestra.run.smithi142.stdout:Reading package lists... 2022-04-02T03:06:31.578 INFO:teuthology.orchestra.run.smithi104.stdout:Reading package lists... 2022-04-02T03:06:31.646 INFO:teuthology.orchestra.run.smithi029.stdout:Building dependency tree... 2022-04-02T03:06:31.646 INFO:teuthology.orchestra.run.smithi029.stdout:Reading state information... 2022-04-02T03:06:31.728 INFO:teuthology.orchestra.run.smithi142.stdout:Building dependency tree... 2022-04-02T03:06:31.728 INFO:teuthology.orchestra.run.smithi142.stdout:Reading state information... 2022-04-02T03:06:31.753 INFO:teuthology.orchestra.run.smithi104.stdout:Building dependency tree... 2022-04-02T03:06:31.754 INFO:teuthology.orchestra.run.smithi104.stdout:Reading state information... 2022-04-02T03:06:31.804 INFO:teuthology.orchestra.run.smithi029.stdout:bridge-utils is already the newest version (1.6-2ubuntu1). 2022-04-02T03:06:31.804 INFO:teuthology.orchestra.run.smithi029.stdout:bridge-utils set to manually installed. 2022-04-02T03:06:31.804 INFO:teuthology.orchestra.run.smithi029.stdout:The following additional packages will be installed: 2022-04-02T03:06:31.805 INFO:teuthology.orchestra.run.smithi029.stdout: conntrack cri-tools ebtables kubernetes-cni socat 2022-04-02T03:06:31.805 INFO:teuthology.orchestra.run.smithi029.stdout:Suggested packages: 2022-04-02T03:06:31.806 INFO:teuthology.orchestra.run.smithi029.stdout: nftables 2022-04-02T03:06:31.857 INFO:teuthology.orchestra.run.smithi029.stdout:The following NEW packages will be installed: 2022-04-02T03:06:31.858 INFO:teuthology.orchestra.run.smithi029.stdout: conntrack cri-tools ebtables kubeadm kubectl kubelet kubernetes-cni socat 2022-04-02T03:06:31.875 INFO:teuthology.orchestra.run.smithi142.stdout:bridge-utils is already the newest version (1.6-2ubuntu1). 2022-04-02T03:06:31.875 INFO:teuthology.orchestra.run.smithi142.stdout:bridge-utils set to manually installed. 2022-04-02T03:06:31.876 INFO:teuthology.orchestra.run.smithi142.stdout:The following additional packages will be installed: 2022-04-02T03:06:31.876 INFO:teuthology.orchestra.run.smithi142.stdout: conntrack cri-tools ebtables kubernetes-cni socat 2022-04-02T03:06:31.877 INFO:teuthology.orchestra.run.smithi142.stdout:Suggested packages: 2022-04-02T03:06:31.877 INFO:teuthology.orchestra.run.smithi142.stdout: nftables 2022-04-02T03:06:31.887 INFO:teuthology.orchestra.run.smithi104.stdout:bridge-utils is already the newest version (1.6-2ubuntu1). 2022-04-02T03:06:31.887 INFO:teuthology.orchestra.run.smithi104.stdout:bridge-utils set to manually installed. 2022-04-02T03:06:31.888 INFO:teuthology.orchestra.run.smithi104.stdout:The following additional packages will be installed: 2022-04-02T03:06:31.888 INFO:teuthology.orchestra.run.smithi104.stdout: conntrack cri-tools ebtables kubernetes-cni socat 2022-04-02T03:06:31.889 INFO:teuthology.orchestra.run.smithi104.stdout:Suggested packages: 2022-04-02T03:06:31.889 INFO:teuthology.orchestra.run.smithi104.stdout: nftables 2022-04-02T03:06:31.925 INFO:teuthology.orchestra.run.smithi142.stdout:The following NEW packages will be installed: 2022-04-02T03:06:31.926 INFO:teuthology.orchestra.run.smithi142.stdout: conntrack cri-tools ebtables kubeadm kubectl kubelet kubernetes-cni socat 2022-04-02T03:06:31.929 INFO:teuthology.orchestra.run.smithi029.stdout:0 upgraded, 8 newly installed, 0 to remove and 109 not upgraded. 2022-04-02T03:06:31.929 INFO:teuthology.orchestra.run.smithi029.stdout:Need to get 77.7 MB of archives. 2022-04-02T03:06:31.930 INFO:teuthology.orchestra.run.smithi029.stdout:After this operation, 335 MB of additional disk space will be used. 2022-04-02T03:06:31.930 INFO:teuthology.orchestra.run.smithi029.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 conntrack amd64 1:1.4.5-2 [30.3 kB] 2022-04-02T03:06:31.936 INFO:teuthology.orchestra.run.smithi104.stdout:The following NEW packages will be installed: 2022-04-02T03:06:31.937 INFO:teuthology.orchestra.run.smithi104.stdout: conntrack cri-tools ebtables kubeadm kubectl kubelet kubernetes-cni socat 2022-04-02T03:06:31.971 INFO:teuthology.orchestra.run.smithi029.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 ebtables amd64 2.0.11-3build1 [80.3 kB] 2022-04-02T03:06:32.001 INFO:teuthology.orchestra.run.smithi029.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2022-04-02T03:06:32.004 INFO:teuthology.orchestra.run.smithi104.stdout:0 upgraded, 8 newly installed, 0 to remove and 109 not upgraded. 2022-04-02T03:06:32.004 INFO:teuthology.orchestra.run.smithi104.stdout:Need to get 77.7 MB of archives. 2022-04-02T03:06:32.004 INFO:teuthology.orchestra.run.smithi104.stdout:After this operation, 335 MB of additional disk space will be used. 2022-04-02T03:06:32.004 INFO:teuthology.orchestra.run.smithi104.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 conntrack amd64 1:1.4.5-2 [30.3 kB] 2022-04-02T03:06:32.050 INFO:teuthology.orchestra.run.smithi104.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 ebtables amd64 2.0.11-3build1 [80.3 kB] 2022-04-02T03:06:32.077 INFO:teuthology.orchestra.run.smithi104.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2022-04-02T03:06:32.114 INFO:teuthology.orchestra.run.smithi142.stdout:0 upgraded, 8 newly installed, 0 to remove and 109 not upgraded. 2022-04-02T03:06:32.114 INFO:teuthology.orchestra.run.smithi142.stdout:Need to get 77.7 MB of archives. 2022-04-02T03:06:32.115 INFO:teuthology.orchestra.run.smithi142.stdout:After this operation, 335 MB of additional disk space will be used. 2022-04-02T03:06:32.115 INFO:teuthology.orchestra.run.smithi142.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 conntrack amd64 1:1.4.5-2 [30.3 kB] 2022-04-02T03:06:32.120 INFO:teuthology.orchestra.run.smithi029.stdout:Get:3 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 cri-tools amd64 1.23.0-00 [15.3 MB] 2022-04-02T03:06:32.166 INFO:teuthology.orchestra.run.smithi142.stdout:Get:1 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 cri-tools amd64 1.23.0-00 [15.3 MB] 2022-04-02T03:06:32.207 INFO:teuthology.orchestra.run.smithi104.stdout:Get:3 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 cri-tools amd64 1.23.0-00 [15.3 MB] 2022-04-02T03:06:32.311 INFO:teuthology.orchestra.run.smithi142.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/main amd64 ebtables amd64 2.0.11-3build1 [80.3 kB] 2022-04-02T03:06:32.459 INFO:teuthology.orchestra.run.smithi142.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2022-04-02T03:06:32.688 INFO:teuthology.orchestra.run.smithi142.stdout:Get:2 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubernetes-cni amd64 0.8.7-00 [25.0 MB] 2022-04-02T03:06:32.731 INFO:teuthology.orchestra.run.smithi029.stdout:Get:5 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubernetes-cni amd64 0.8.7-00 [25.0 MB] 2022-04-02T03:06:33.010 INFO:teuthology.orchestra.run.smithi104.stdout:Get:5 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubernetes-cni amd64 0.8.7-00 [25.0 MB] 2022-04-02T03:06:33.330 INFO:teuthology.orchestra.run.smithi142.stdout:Get:3 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubelet amd64 1.23.5-00 [19.5 MB] 2022-04-02T03:06:33.773 INFO:teuthology.orchestra.run.smithi142.stdout:Get:5 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubectl amd64 1.23.5-00 [8,931 kB] 2022-04-02T03:06:33.843 INFO:teuthology.orchestra.run.smithi104.stdout:Get:6 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubelet amd64 1.23.5-00 [19.5 MB] 2022-04-02T03:06:33.922 INFO:teuthology.orchestra.run.smithi142.stdout:Get:6 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubeadm amd64 1.23.5-00 [8,582 kB] 2022-04-02T03:06:34.263 INFO:teuthology.orchestra.run.smithi104.stdout:Get:7 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubectl amd64 1.23.5-00 [8,931 kB] 2022-04-02T03:06:34.322 INFO:teuthology.orchestra.run.smithi142.stderr:debconf: unable to initialize frontend: Dialog 2022-04-02T03:06:34.323 INFO:teuthology.orchestra.run.smithi142.stderr:debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) 2022-04-02T03:06:34.323 INFO:teuthology.orchestra.run.smithi142.stderr:debconf: falling back to frontend: Readline 2022-04-02T03:06:34.331 INFO:teuthology.orchestra.run.smithi142.stderr:debconf: unable to initialize frontend: Readline 2022-04-02T03:06:34.331 INFO:teuthology.orchestra.run.smithi142.stderr:debconf: (This frontend requires a controlling tty.) 2022-04-02T03:06:34.331 INFO:teuthology.orchestra.run.smithi142.stderr:debconf: falling back to frontend: Teletype 2022-04-02T03:06:34.337 INFO:teuthology.orchestra.run.smithi142.stderr:dpkg-preconfigure: unable to re-open stdin: 2022-04-02T03:06:34.376 INFO:teuthology.orchestra.run.smithi104.stdout:Get:8 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubeadm amd64 1.23.5-00 [8,582 kB] 2022-04-02T03:06:34.387 INFO:teuthology.orchestra.run.smithi142.stdout:Fetched 77.7 MB in 2s (35.8 MB/s) 2022-04-02T03:06:34.454 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package conntrack. 2022-04-02T03:06:34.496 INFO:teuthology.orchestra.run.smithi142.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109244 files and directories currently installed.) 2022-04-02T03:06:34.499 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../0-conntrack_1%3a1.4.5-2_amd64.deb ... 2022-04-02T03:06:34.501 INFO:teuthology.orchestra.run.smithi029.stdout:Get:6 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubelet amd64 1.23.5-00 [19.5 MB] 2022-04-02T03:06:34.536 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking conntrack (1:1.4.5-2) ... 2022-04-02T03:06:34.722 INFO:teuthology.orchestra.run.smithi104.stderr:debconf: unable to initialize frontend: Dialog 2022-04-02T03:06:34.722 INFO:teuthology.orchestra.run.smithi104.stderr:debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) 2022-04-02T03:06:34.722 INFO:teuthology.orchestra.run.smithi104.stderr:debconf: falling back to frontend: Readline 2022-04-02T03:06:34.730 INFO:teuthology.orchestra.run.smithi104.stderr:debconf: unable to initialize frontend: Readline 2022-04-02T03:06:34.730 INFO:teuthology.orchestra.run.smithi104.stderr:debconf: (This frontend requires a controlling tty.) 2022-04-02T03:06:34.731 INFO:teuthology.orchestra.run.smithi104.stderr:debconf: falling back to frontend: Teletype 2022-04-02T03:06:34.736 INFO:teuthology.orchestra.run.smithi104.stderr:dpkg-preconfigure: unable to re-open stdin: 2022-04-02T03:06:34.787 INFO:teuthology.orchestra.run.smithi104.stdout:Fetched 77.7 MB in 3s (30.3 MB/s) 2022-04-02T03:06:34.839 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package cri-tools. 2022-04-02T03:06:34.852 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../1-cri-tools_1.23.0-00_amd64.deb ... 2022-04-02T03:06:34.880 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package conntrack. 2022-04-02T03:06:34.889 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking cri-tools (1.23.0-00) ... 2022-04-02T03:06:34.919 INFO:teuthology.orchestra.run.smithi104.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109244 files and directories currently installed.) 2022-04-02T03:06:34.923 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../0-conntrack_1%3a1.4.5-2_amd64.deb ... 2022-04-02T03:06:34.963 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking conntrack (1:1.4.5-2) ... 2022-04-02T03:06:35.266 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package cri-tools. 2022-04-02T03:06:35.279 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../1-cri-tools_1.23.0-00_amd64.deb ... 2022-04-02T03:06:35.306 INFO:teuthology.orchestra.run.smithi029.stdout:Get:7 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubectl amd64 1.23.5-00 [8,931 kB] 2022-04-02T03:06:35.324 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking cri-tools (1.23.0-00) ... 2022-04-02T03:06:35.649 INFO:teuthology.orchestra.run.smithi029.stdout:Get:8 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubeadm amd64 1.23.5-00 [8,582 kB] 2022-04-02T03:06:36.151 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package ebtables. 2022-04-02T03:06:36.163 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../2-ebtables_2.0.11-3build1_amd64.deb ... 2022-04-02T03:06:36.178 INFO:teuthology.orchestra.run.smithi029.stderr:debconf: unable to initialize frontend: Dialog 2022-04-02T03:06:36.178 INFO:teuthology.orchestra.run.smithi029.stderr:debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) 2022-04-02T03:06:36.178 INFO:teuthology.orchestra.run.smithi029.stderr:debconf: falling back to frontend: Readline 2022-04-02T03:06:36.190 INFO:teuthology.orchestra.run.smithi029.stderr:debconf: unable to initialize frontend: Readline 2022-04-02T03:06:36.190 INFO:teuthology.orchestra.run.smithi029.stderr:debconf: (This frontend requires a controlling tty.) 2022-04-02T03:06:36.190 INFO:teuthology.orchestra.run.smithi029.stderr:debconf: falling back to frontend: Teletype 2022-04-02T03:06:36.196 INFO:teuthology.orchestra.run.smithi029.stderr:dpkg-preconfigure: unable to re-open stdin: 2022-04-02T03:06:36.208 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking ebtables (2.0.11-3build1) ... 2022-04-02T03:06:36.244 INFO:teuthology.orchestra.run.smithi029.stdout:Fetched 77.7 MB in 4s (19.0 MB/s) 2022-04-02T03:06:36.325 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package conntrack. 2022-04-02T03:06:36.367 INFO:teuthology.orchestra.run.smithi029.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109244 files and directories currently installed.) 2022-04-02T03:06:36.371 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../0-conntrack_1%3a1.4.5-2_amd64.deb ... 2022-04-02T03:06:36.415 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking conntrack (1:1.4.5-2) ... 2022-04-02T03:06:36.470 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package kubernetes-cni. 2022-04-02T03:06:36.484 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../3-kubernetes-cni_0.8.7-00_amd64.deb ... 2022-04-02T03:06:36.528 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking kubernetes-cni (0.8.7-00) ... 2022-04-02T03:06:36.636 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package ebtables. 2022-04-02T03:06:36.650 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../2-ebtables_2.0.11-3build1_amd64.deb ... 2022-04-02T03:06:36.710 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking ebtables (2.0.11-3build1) ... 2022-04-02T03:06:36.730 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package cri-tools. 2022-04-02T03:06:36.744 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../1-cri-tools_1.23.0-00_amd64.deb ... 2022-04-02T03:06:36.789 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking cri-tools (1.23.0-00) ... 2022-04-02T03:06:36.974 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package kubernetes-cni. 2022-04-02T03:06:36.986 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../3-kubernetes-cni_0.8.7-00_amd64.deb ... 2022-04-02T03:06:37.024 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking kubernetes-cni (0.8.7-00) ... 2022-04-02T03:06:37.915 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package socat. 2022-04-02T03:06:37.929 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../4-socat_1.7.3.3-2_amd64.deb ... 2022-04-02T03:06:37.999 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking socat (1.7.3.3-2) ... 2022-04-02T03:06:38.378 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package kubelet. 2022-04-02T03:06:38.393 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../5-kubelet_1.23.5-00_amd64.deb ... 2022-04-02T03:06:38.427 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking kubelet (1.23.5-00) ... 2022-04-02T03:06:38.451 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package socat. 2022-04-02T03:06:38.454 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package ebtables. 2022-04-02T03:06:38.464 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../4-socat_1.7.3.3-2_amd64.deb ... 2022-04-02T03:06:38.469 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../2-ebtables_2.0.11-3build1_amd64.deb ... 2022-04-02T03:06:38.501 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking socat (1.7.3.3-2) ... 2022-04-02T03:06:38.530 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking ebtables (2.0.11-3build1) ... 2022-04-02T03:06:38.810 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package kubernetes-cni. 2022-04-02T03:06:38.824 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../3-kubernetes-cni_0.8.7-00_amd64.deb ... 2022-04-02T03:06:38.868 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking kubernetes-cni (0.8.7-00) ... 2022-04-02T03:06:38.912 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package kubelet. 2022-04-02T03:06:38.925 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../5-kubelet_1.23.5-00_amd64.deb ... 2022-04-02T03:06:38.962 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking kubelet (1.23.5-00) ... 2022-04-02T03:06:40.367 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package socat. 2022-04-02T03:06:40.380 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../4-socat_1.7.3.3-2_amd64.deb ... 2022-04-02T03:06:40.417 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking socat (1.7.3.3-2) ... 2022-04-02T03:06:40.788 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package kubelet. 2022-04-02T03:06:40.801 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../5-kubelet_1.23.5-00_amd64.deb ... 2022-04-02T03:06:40.804 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package kubectl. 2022-04-02T03:06:40.818 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../6-kubectl_1.23.5-00_amd64.deb ... 2022-04-02T03:06:40.838 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking kubelet (1.23.5-00) ... 2022-04-02T03:06:40.863 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking kubectl (1.23.5-00) ... 2022-04-02T03:06:41.334 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package kubectl. 2022-04-02T03:06:41.348 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../6-kubectl_1.23.5-00_amd64.deb ... 2022-04-02T03:06:41.393 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking kubectl (1.23.5-00) ... 2022-04-02T03:06:41.990 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package kubeadm. 2022-04-02T03:06:42.005 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../7-kubeadm_1.23.5-00_amd64.deb ... 2022-04-02T03:06:42.040 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking kubeadm (1.23.5-00) ... 2022-04-02T03:06:42.557 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package kubeadm. 2022-04-02T03:06:42.570 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../7-kubeadm_1.23.5-00_amd64.deb ... 2022-04-02T03:06:42.615 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking kubeadm (1.23.5-00) ... 2022-04-02T03:06:43.164 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up conntrack (1:1.4.5-2) ... 2022-04-02T03:06:43.281 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up kubectl (1.23.5-00) ... 2022-04-02T03:06:43.400 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up ebtables (2.0.11-3build1) ... 2022-04-02T03:06:43.758 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up conntrack (1:1.4.5-2) ... 2022-04-02T03:06:43.875 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up kubectl (1.23.5-00) ... 2022-04-02T03:06:44.012 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up ebtables (2.0.11-3build1) ... 2022-04-02T03:06:44.077 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package kubectl. 2022-04-02T03:06:44.091 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../6-kubectl_1.23.5-00_amd64.deb ... 2022-04-02T03:06:44.127 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking kubectl (1.23.5-00) ... 2022-04-02T03:06:44.190 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up socat (1.7.3.3-2) ... 2022-04-02T03:06:44.308 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up cri-tools (1.23.0-00) ... 2022-04-02T03:06:44.438 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up kubernetes-cni (0.8.7-00) ... 2022-04-02T03:06:44.565 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up kubelet (1.23.5-00) ... 2022-04-02T03:06:44.784 INFO:teuthology.orchestra.run.smithi142.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/kubelet.service -> /lib/systemd/system/kubelet.service. 2022-04-02T03:06:44.845 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up socat (1.7.3.3-2) ... 2022-04-02T03:06:44.973 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up cri-tools (1.23.0-00) ... 2022-04-02T03:06:45.093 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up kubernetes-cni (0.8.7-00) ... 2022-04-02T03:06:45.219 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up kubelet (1.23.5-00) ... 2022-04-02T03:06:45.321 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up kubeadm (1.23.5-00) ... 2022-04-02T03:06:45.437 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/kubelet.service -> /lib/systemd/system/kubelet.service. 2022-04-02T03:06:45.656 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package kubeadm. 2022-04-02T03:06:45.671 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../7-kubeadm_1.23.5-00_amd64.deb ... 2022-04-02T03:06:45.708 INFO:teuthology.orchestra.run.smithi142.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-04-02T03:06:45.715 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking kubeadm (1.23.5-00) ... 2022-04-02T03:06:45.938 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up kubeadm (1.23.5-00) ... 2022-04-02T03:06:46.351 INFO:teuthology.orchestra.run.smithi104.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-04-02T03:06:47.158 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up conntrack (1:1.4.5-2) ... 2022-04-02T03:06:47.283 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up kubectl (1.23.5-00) ... 2022-04-02T03:06:47.415 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up ebtables (2.0.11-3build1) ... 2022-04-02T03:06:48.254 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up socat (1.7.3.3-2) ... 2022-04-02T03:06:48.385 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up cri-tools (1.23.0-00) ... 2022-04-02T03:06:48.500 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up kubernetes-cni (0.8.7-00) ... 2022-04-02T03:06:48.633 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up kubelet (1.23.5-00) ... 2022-04-02T03:06:48.848 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/kubelet.service -> /lib/systemd/system/kubelet.service. 2022-04-02T03:06:49.361 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up kubeadm (1.23.5-00) ... 2022-04-02T03:06:49.739 INFO:teuthology.orchestra.run.smithi029.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-04-02T03:06:50.621 DEBUG:teuthology.orchestra.run.smithi029:> sudo systemctl enable --now kubelet && sudo kubeadm config images pull 2022-04-02T03:06:50.624 DEBUG:teuthology.orchestra.run.smithi104:> sudo systemctl enable --now kubelet && sudo kubeadm config images pull 2022-04-02T03:06:50.627 DEBUG:teuthology.orchestra.run.smithi142:> sudo systemctl enable --now kubelet && sudo kubeadm config images pull 2022-04-02T03:06:59.987 INFO:teuthology.orchestra.run.smithi142.stdout:[config/images] Pulled k8s.gcr.io/kube-apiserver:v1.23.5 2022-04-02T03:07:00.588 INFO:teuthology.orchestra.run.smithi104.stdout:[config/images] Pulled k8s.gcr.io/kube-apiserver:v1.23.5 2022-04-02T03:07:03.009 INFO:teuthology.orchestra.run.smithi142.stdout:[config/images] Pulled k8s.gcr.io/kube-controller-manager:v1.23.5 2022-04-02T03:07:03.400 INFO:teuthology.orchestra.run.smithi104.stdout:[config/images] Pulled k8s.gcr.io/kube-controller-manager:v1.23.5 2022-04-02T03:07:05.222 INFO:teuthology.orchestra.run.smithi142.stdout:[config/images] Pulled k8s.gcr.io/kube-scheduler:v1.23.5 2022-04-02T03:07:05.275 INFO:teuthology.orchestra.run.smithi029.stdout:[config/images] Pulled k8s.gcr.io/kube-apiserver:v1.23.5 2022-04-02T03:07:05.505 INFO:teuthology.orchestra.run.smithi104.stdout:[config/images] Pulled k8s.gcr.io/kube-scheduler:v1.23.5 2022-04-02T03:07:08.452 INFO:teuthology.orchestra.run.smithi029.stdout:[config/images] Pulled k8s.gcr.io/kube-controller-manager:v1.23.5 2022-04-02T03:07:08.522 INFO:teuthology.orchestra.run.smithi142.stdout:[config/images] Pulled k8s.gcr.io/kube-proxy:v1.23.5 2022-04-02T03:07:09.853 INFO:teuthology.orchestra.run.smithi104.stdout:[config/images] Pulled k8s.gcr.io/kube-proxy:v1.23.5 2022-04-02T03:07:09.871 INFO:teuthology.orchestra.run.smithi142.stdout:[config/images] Pulled k8s.gcr.io/pause:3.6 2022-04-02T03:07:10.549 INFO:teuthology.orchestra.run.smithi029.stdout:[config/images] Pulled k8s.gcr.io/kube-scheduler:v1.23.5 2022-04-02T03:07:11.186 INFO:teuthology.orchestra.run.smithi104.stdout:[config/images] Pulled k8s.gcr.io/pause:3.6 2022-04-02T03:07:14.649 INFO:teuthology.orchestra.run.smithi029.stdout:[config/images] Pulled k8s.gcr.io/kube-proxy:v1.23.5 2022-04-02T03:07:15.104 INFO:teuthology.orchestra.run.smithi142.stdout:[config/images] Pulled k8s.gcr.io/etcd:3.5.1-0 2022-04-02T03:07:16.063 INFO:teuthology.orchestra.run.smithi029.stdout:[config/images] Pulled k8s.gcr.io/pause:3.6 2022-04-02T03:07:16.835 INFO:teuthology.orchestra.run.smithi104.stdout:[config/images] Pulled k8s.gcr.io/etcd:3.5.1-0 2022-04-02T03:07:17.229 INFO:teuthology.orchestra.run.smithi142.stdout:[config/images] Pulled k8s.gcr.io/coredns/coredns:v1.8.6 2022-04-02T03:07:19.043 INFO:teuthology.orchestra.run.smithi104.stdout:[config/images] Pulled k8s.gcr.io/coredns/coredns:v1.8.6 2022-04-02T03:07:21.415 INFO:teuthology.orchestra.run.smithi029.stdout:[config/images] Pulled k8s.gcr.io/etcd:3.5.1-0 2022-04-02T03:07:23.873 INFO:teuthology.orchestra.run.smithi029.stdout:[config/images] Pulled k8s.gcr.io/coredns/coredns:v1.8.6 2022-04-02T03:07:23.878 INFO:tasks.kubeadm:Token: abcdef.l0uwoavaalqnqh3t 2022-04-02T03:07:23.878 INFO:tasks.kubeadm:Remotes: {Remote(name='ubuntu@smithi029.front.sepia.ceph.com'): '172.21.15.29', Remote(name='ubuntu@smithi104.front.sepia.ceph.com'): '172.21.15.104', Remote(name='ubuntu@smithi142.front.sepia.ceph.com'): '172.21.15.142'} 2022-04-02T03:07:23.879 DEBUG:teuthology.orchestra.run.smithi029:> sudo kubeadm init --node-name smithi029 --token abcdef.l0uwoavaalqnqh3t --pod-network-cidr 10.248.224.0/21 2022-04-02T03:07:24.072 INFO:teuthology.orchestra.run.smithi029.stdout:[init] Using Kubernetes version: v1.23.5 2022-04-02T03:07:24.073 INFO:teuthology.orchestra.run.smithi029.stdout:[preflight] Running pre-flight checks 2022-04-02T03:07:24.521 INFO:teuthology.orchestra.run.smithi029.stdout:[preflight] Pulling images required for setting up a Kubernetes cluster 2022-04-02T03:07:24.522 INFO:teuthology.orchestra.run.smithi029.stdout:[preflight] This might take a minute or two, depending on the speed of your internet connection 2022-04-02T03:07:24.523 INFO:teuthology.orchestra.run.smithi029.stdout:[preflight] You can also perform this action in beforehand using 'kubeadm config images pull' 2022-04-02T03:07:24.835 INFO:teuthology.orchestra.run.smithi029.stdout:[certs] Using certificateDir folder "/etc/kubernetes/pki" 2022-04-02T03:07:25.116 INFO:teuthology.orchestra.run.smithi029.stdout:[certs] Generating "ca" certificate and key 2022-04-02T03:07:25.334 INFO:teuthology.orchestra.run.smithi029.stdout:[certs] Generating "apiserver" certificate and key 2022-04-02T03:07:25.334 INFO:teuthology.orchestra.run.smithi029.stdout:[certs] apiserver serving cert is signed for DNS names [kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local smithi029] and IPs [10.96.0.1 172.21.15.29] 2022-04-02T03:07:25.529 INFO:teuthology.orchestra.run.smithi029.stdout:[certs] Generating "apiserver-kubelet-client" certificate and key 2022-04-02T03:07:25.733 INFO:teuthology.orchestra.run.smithi029.stdout:[certs] Generating "front-proxy-ca" certificate and key 2022-04-02T03:07:25.955 INFO:teuthology.orchestra.run.smithi029.stdout:[certs] Generating "front-proxy-client" certificate and key 2022-04-02T03:07:26.264 INFO:teuthology.orchestra.run.smithi029.stdout:[certs] Generating "etcd/ca" certificate and key 2022-04-02T03:07:26.743 INFO:teuthology.orchestra.run.smithi029.stdout:[certs] Generating "etcd/server" certificate and key 2022-04-02T03:07:26.743 INFO:teuthology.orchestra.run.smithi029.stdout:[certs] etcd/server serving cert is signed for DNS names [localhost smithi029] and IPs [172.21.15.29 127.0.0.1 ::1] 2022-04-02T03:07:26.866 INFO:teuthology.orchestra.run.smithi029.stdout:[certs] Generating "etcd/peer" certificate and key 2022-04-02T03:07:26.866 INFO:teuthology.orchestra.run.smithi029.stdout:[certs] etcd/peer serving cert is signed for DNS names [localhost smithi029] and IPs [172.21.15.29 127.0.0.1 ::1] 2022-04-02T03:07:27.120 INFO:teuthology.orchestra.run.smithi029.stdout:[certs] Generating "etcd/healthcheck-client" certificate and key 2022-04-02T03:07:27.608 INFO:teuthology.orchestra.run.smithi029.stdout:[certs] Generating "apiserver-etcd-client" certificate and key 2022-04-02T03:07:28.067 INFO:teuthology.orchestra.run.smithi029.stdout:[certs] Generating "sa" key and public key 2022-04-02T03:07:28.068 INFO:teuthology.orchestra.run.smithi029.stdout:[kubeconfig] Using kubeconfig folder "/etc/kubernetes" 2022-04-02T03:07:28.497 INFO:teuthology.orchestra.run.smithi029.stdout:[kubeconfig] Writing "admin.conf" kubeconfig file 2022-04-02T03:07:28.641 INFO:teuthology.orchestra.run.smithi029.stdout:[kubeconfig] Writing "kubelet.conf" kubeconfig file 2022-04-02T03:07:28.796 INFO:teuthology.orchestra.run.smithi029.stdout:[kubeconfig] Writing "controller-manager.conf" kubeconfig file 2022-04-02T03:07:29.079 INFO:teuthology.orchestra.run.smithi029.stdout:[kubeconfig] Writing "scheduler.conf" kubeconfig file 2022-04-02T03:07:29.095 INFO:teuthology.orchestra.run.smithi029.stdout:[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" 2022-04-02T03:07:29.097 INFO:teuthology.orchestra.run.smithi029.stdout:[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" 2022-04-02T03:07:29.097 INFO:teuthology.orchestra.run.smithi029.stdout:[kubelet-start] Starting the kubelet 2022-04-02T03:07:29.307 INFO:teuthology.orchestra.run.smithi029.stdout:[control-plane] Using manifest folder "/etc/kubernetes/manifests" 2022-04-02T03:07:29.307 INFO:teuthology.orchestra.run.smithi029.stdout:[control-plane] Creating static Pod manifest for "kube-apiserver" 2022-04-02T03:07:29.311 INFO:teuthology.orchestra.run.smithi029.stdout:[control-plane] Creating static Pod manifest for "kube-controller-manager" 2022-04-02T03:07:29.312 INFO:teuthology.orchestra.run.smithi029.stdout:[control-plane] Creating static Pod manifest for "kube-scheduler" 2022-04-02T03:07:29.313 INFO:teuthology.orchestra.run.smithi029.stdout:[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests" 2022-04-02T03:07:29.314 INFO:teuthology.orchestra.run.smithi029.stdout:[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s 2022-04-02T03:07:52.818 INFO:teuthology.orchestra.run.smithi029.stdout:[apiclient] All control plane components are healthy after 23.503427 seconds 2022-04-02T03:07:52.819 INFO:teuthology.orchestra.run.smithi029.stdout:[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace 2022-04-02T03:07:52.837 INFO:teuthology.orchestra.run.smithi029.stdout:[kubelet] Creating a ConfigMap "kubelet-config-1.23" in namespace kube-system with the configuration for the kubelets in the cluster 2022-04-02T03:07:52.837 INFO:teuthology.orchestra.run.smithi029.stdout:NOTE: The "kubelet-config-1.23" naming of the kubelet ConfigMap is deprecated. Once the UnversionedKubeletConfigMap feature gate graduates to Beta the default name will become just "kubelet-config". Kubeadm upgrade will handle this transition transparently. 2022-04-02T03:07:53.446 INFO:teuthology.orchestra.run.smithi029.stdout:[upload-certs] Skipping phase. Please see --upload-certs 2022-04-02T03:07:53.447 INFO:teuthology.orchestra.run.smithi029.stdout:[mark-control-plane] Marking the node smithi029 as control-plane by adding the labels: [node-role.kubernetes.io/master(deprecated) node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers] 2022-04-02T03:07:53.447 INFO:teuthology.orchestra.run.smithi029.stdout:[mark-control-plane] Marking the node smithi029 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule] 2022-04-02T03:07:53.973 INFO:teuthology.orchestra.run.smithi029.stdout:[bootstrap-token] Using token: abcdef.l0uwoavaalqnqh3t 2022-04-02T03:07:53.973 INFO:teuthology.orchestra.run.smithi029.stdout:[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles 2022-04-02T03:07:53.982 INFO:teuthology.orchestra.run.smithi029.stdout:[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes 2022-04-02T03:07:54.012 INFO:teuthology.orchestra.run.smithi029.stdout:[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials 2022-04-02T03:07:54.021 INFO:teuthology.orchestra.run.smithi029.stdout:[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token 2022-04-02T03:07:54.031 INFO:teuthology.orchestra.run.smithi029.stdout:[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster 2022-04-02T03:07:54.050 INFO:teuthology.orchestra.run.smithi029.stdout:[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace 2022-04-02T03:07:54.128 INFO:teuthology.orchestra.run.smithi029.stdout:[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key 2022-04-02T03:07:54.614 INFO:teuthology.orchestra.run.smithi029.stdout:[addons] Applied essential addon: CoreDNS 2022-04-02T03:07:54.868 INFO:teuthology.orchestra.run.smithi029.stdout:[addons] Applied essential addon: kube-proxy 2022-04-02T03:07:54.870 INFO:teuthology.orchestra.run.smithi029.stdout: 2022-04-02T03:07:54.870 INFO:teuthology.orchestra.run.smithi029.stdout:Your Kubernetes control-plane has initialized successfully! 2022-04-02T03:07:54.871 INFO:teuthology.orchestra.run.smithi029.stdout: 2022-04-02T03:07:54.871 INFO:teuthology.orchestra.run.smithi029.stdout:To start using your cluster, you need to run the following as a regular user: 2022-04-02T03:07:54.871 INFO:teuthology.orchestra.run.smithi029.stdout: 2022-04-02T03:07:54.871 INFO:teuthology.orchestra.run.smithi029.stdout: mkdir -p $HOME/.kube 2022-04-02T03:07:54.871 INFO:teuthology.orchestra.run.smithi029.stdout: sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config 2022-04-02T03:07:54.872 INFO:teuthology.orchestra.run.smithi029.stdout: sudo chown $(id -u):$(id -g) $HOME/.kube/config 2022-04-02T03:07:54.872 INFO:teuthology.orchestra.run.smithi029.stdout: 2022-04-02T03:07:54.872 INFO:teuthology.orchestra.run.smithi029.stdout:Alternatively, if you are the root user, you can run: 2022-04-02T03:07:54.872 INFO:teuthology.orchestra.run.smithi029.stdout: 2022-04-02T03:07:54.873 INFO:teuthology.orchestra.run.smithi029.stdout: export KUBECONFIG=/etc/kubernetes/admin.conf 2022-04-02T03:07:54.873 INFO:teuthology.orchestra.run.smithi029.stdout: 2022-04-02T03:07:54.873 INFO:teuthology.orchestra.run.smithi029.stdout:You should now deploy a pod network to the cluster. 2022-04-02T03:07:54.873 INFO:teuthology.orchestra.run.smithi029.stdout:Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at: 2022-04-02T03:07:54.873 INFO:teuthology.orchestra.run.smithi029.stdout: https://kubernetes.io/docs/concepts/cluster-administration/addons/ 2022-04-02T03:07:54.874 INFO:teuthology.orchestra.run.smithi029.stdout: 2022-04-02T03:07:54.874 INFO:teuthology.orchestra.run.smithi029.stdout:Then you can join any number of worker nodes by running the following on each as root: 2022-04-02T03:07:54.875 INFO:teuthology.orchestra.run.smithi029.stdout: 2022-04-02T03:07:54.875 INFO:teuthology.orchestra.run.smithi029.stdout:kubeadm join 172.21.15.29:6443 --token abcdef.l0uwoavaalqnqh3t \ 2022-04-02T03:07:54.875 INFO:teuthology.orchestra.run.smithi029.stdout: --discovery-token-ca-cert-hash sha256:f12beacdb4b81c9fbdf376347deff1de6686873c2f2f0374ec0298a46a06ebdd 2022-04-02T03:07:54.877 DEBUG:teuthology.orchestra.run.smithi104:> sudo kubeadm join 172.21.15.29:6443 --node-name smithi104 --token abcdef.l0uwoavaalqnqh3t --discovery-token-unsafe-skip-ca-verification 2022-04-02T03:07:54.881 DEBUG:teuthology.orchestra.run.smithi142:> sudo kubeadm join 172.21.15.29:6443 --node-name smithi142 --token abcdef.l0uwoavaalqnqh3t --discovery-token-unsafe-skip-ca-verification 2022-04-02T03:07:54.920 INFO:teuthology.orchestra.run.smithi104.stdout:[preflight] Running pre-flight checks 2022-04-02T03:07:54.923 INFO:teuthology.orchestra.run.smithi142.stdout:[preflight] Running pre-flight checks 2022-04-02T03:08:07.667 INFO:teuthology.orchestra.run.smithi142.stdout:[preflight] Reading configuration from the cluster... 2022-04-02T03:08:07.668 INFO:teuthology.orchestra.run.smithi142.stdout:[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml' 2022-04-02T03:08:07.668 INFO:teuthology.orchestra.run.smithi104.stdout:[preflight] Reading configuration from the cluster... 2022-04-02T03:08:07.668 INFO:teuthology.orchestra.run.smithi104.stdout:[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml' 2022-04-02T03:08:08.090 INFO:teuthology.orchestra.run.smithi104.stderr:W0402 03:08:08.091266 11149 utils.go:69] The recommended value for "resolvConf" in "KubeletConfiguration" is: /run/systemd/resolve/resolv.conf; the provided value is: /run/systemd/resolve/resolv.conf 2022-04-02T03:08:08.091 INFO:teuthology.orchestra.run.smithi142.stderr:W0402 03:08:08.092335 11245 utils.go:69] The recommended value for "resolvConf" in "KubeletConfiguration" is: /run/systemd/resolve/resolv.conf; the provided value is: /run/systemd/resolve/resolv.conf 2022-04-02T03:08:08.107 INFO:teuthology.orchestra.run.smithi142.stdout:[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" 2022-04-02T03:08:08.108 INFO:teuthology.orchestra.run.smithi142.stdout:[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" 2022-04-02T03:08:08.108 INFO:teuthology.orchestra.run.smithi142.stdout:[kubelet-start] Starting the kubelet 2022-04-02T03:08:08.109 INFO:teuthology.orchestra.run.smithi104.stdout:[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" 2022-04-02T03:08:08.109 INFO:teuthology.orchestra.run.smithi104.stdout:[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" 2022-04-02T03:08:08.109 INFO:teuthology.orchestra.run.smithi104.stdout:[kubelet-start] Starting the kubelet 2022-04-02T03:08:08.341 INFO:teuthology.orchestra.run.smithi142.stdout:[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap... 2022-04-02T03:08:08.342 INFO:teuthology.orchestra.run.smithi104.stdout:[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap... 2022-04-02T03:08:13.879 INFO:teuthology.orchestra.run.smithi104.stdout: 2022-04-02T03:08:13.879 INFO:teuthology.orchestra.run.smithi104.stdout:This node has joined the cluster: 2022-04-02T03:08:13.880 INFO:teuthology.orchestra.run.smithi104.stdout:* Certificate signing request was sent to apiserver and a response was received. 2022-04-02T03:08:13.880 INFO:teuthology.orchestra.run.smithi104.stdout:* The Kubelet was informed of the new secure connection details. 2022-04-02T03:08:13.880 INFO:teuthology.orchestra.run.smithi104.stdout: 2022-04-02T03:08:13.880 INFO:teuthology.orchestra.run.smithi104.stdout:Run 'kubectl get nodes' on the control-plane to see this node join the cluster. 2022-04-02T03:08:13.881 INFO:teuthology.orchestra.run.smithi104.stdout: 2022-04-02T03:08:13.906 INFO:teuthology.orchestra.run.smithi142.stdout: 2022-04-02T03:08:13.906 INFO:teuthology.orchestra.run.smithi142.stdout:This node has joined the cluster: 2022-04-02T03:08:13.906 INFO:teuthology.orchestra.run.smithi142.stdout:* Certificate signing request was sent to apiserver and a response was received. 2022-04-02T03:08:13.907 INFO:teuthology.orchestra.run.smithi142.stdout:* The Kubelet was informed of the new secure connection details. 2022-04-02T03:08:13.907 INFO:teuthology.orchestra.run.smithi142.stdout: 2022-04-02T03:08:13.907 INFO:teuthology.orchestra.run.smithi142.stdout:Run 'kubectl get nodes' on the control-plane to see this node join the cluster. 2022-04-02T03:08:13.907 INFO:teuthology.orchestra.run.smithi142.stdout: 2022-04-02T03:08:13.913 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2022-04-02T03:08:13.914 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/etc/kubernetes/admin.conf of=/dev/stdout 2022-04-02T03:08:13.924 INFO:tasks.kubeadm:Setting up kubectl 2022-04-02T03:08:13.925 DEBUG:teuthology.orchestra.run.smithi029:> mkdir -p .kube && sudo mkdir -p /root/.kube 2022-04-02T03:08:13.937 DEBUG:teuthology.orchestra.run.smithi104:> mkdir -p .kube && sudo mkdir -p /root/.kube 2022-04-02T03:08:13.956 DEBUG:teuthology.orchestra.run.smithi142:> mkdir -p .kube && sudo mkdir -p /root/.kube 2022-04-02T03:08:13.973 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2022-04-02T03:08:13.973 DEBUG:teuthology.orchestra.run.smithi029:> dd of=.kube/config 2022-04-02T03:08:13.981 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2022-04-02T03:08:13.981 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/root/.kube/config 2022-04-02T03:08:14.038 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2022-04-02T03:08:14.038 DEBUG:teuthology.orchestra.run.smithi104:> dd of=.kube/config 2022-04-02T03:08:14.046 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2022-04-02T03:08:14.046 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/root/.kube/config 2022-04-02T03:08:14.062 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2022-04-02T03:08:14.063 DEBUG:teuthology.orchestra.run.smithi142:> dd of=.kube/config 2022-04-02T03:08:14.070 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2022-04-02T03:08:14.071 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd of=/root/.kube/config 2022-04-02T03:08:14.083 DEBUG:teuthology.orchestra.run.smithi029:> curl https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml 2022-04-02T03:08:14.100 INFO:teuthology.orchestra.run.smithi029.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2022-04-02T03:08:14.100 INFO:teuthology.orchestra.run.smithi029.stderr: Dload Upload Total Spent Left Speed 2022-04-02T03:08:14.195 INFO:teuthology.orchestra.run.smithi029.stdout:--- 2022-04-02T03:08:14.195 INFO:teuthology.orchestra.run.smithi029.stdout:apiVersion: policy/v1beta1 2022-04-02T03:08:14.196 INFO:teuthology.orchestra.run.smithi029.stdout:kind: PodSecurityPolicy 2022-04-02T03:08:14.196 INFO:teuthology.orchestra.run.smithi029.stdout:metadata: 2022-04-02T03:08:14.196 INFO:teuthology.orchestra.run.smithi029.stdout: name: psp.flannel.unprivileged 2022-04-02T03:08:14.196 INFO:teuthology.orchestra.run.smithi029.stdout: annotations: 2022-04-02T03:08:14.197 INFO:teuthology.orchestra.run.smithi029.stdout: seccomp.security.alpha.kubernetes.io/allowedProfileNames: docker/default 2022-04-02T03:08:14.197 INFO:teuthology.orchestra.run.smithi029.stdout: seccomp.security.alpha.kubernetes.io/defaultProfileName: docker/default 2022-04-02T03:08:14.197 INFO:teuthology.orchestra.run.smithi029.stdout: apparmor.security.beta.kubernetes.io/allowedProfileNames: runtime/default 2022-04-02T03:08:14.198 INFO:teuthology.orchestra.run.smithi029.stdout: apparmor.security.beta.kubernetes.io/defaultProfileName: runtime/default 2022-04-02T03:08:14.198 INFO:teuthology.orchestra.run.smithi029.stdout:spec: 2022-04-02T03:08:14.198 INFO:teuthology.orchestra.run.smithi029.stdout: privileged: false 2022-04-02T03:08:14.198 INFO:teuthology.orchestra.run.smithi029.stdout: volumes: 2022-04-02T03:08:14.199 INFO:teuthology.orchestra.run.smithi029.stdout: - configMap 2022-04-02T03:08:14.199 INFO:teuthology.orchestra.run.smithi029.stdout: - secret 2022-04-02T03:08:14.199 INFO:teuthology.orchestra.run.smithi029.stdout: - emptyDir 2022-04-02T03:08:14.199 INFO:teuthology.orchestra.run.smithi029.stdout: - hostPath 2022-04-02T03:08:14.200 INFO:teuthology.orchestra.run.smithi029.stdout: allowedHostPaths: 2022-04-02T03:08:14.200 INFO:teuthology.orchestra.run.smithi029.stdout: - pathPrefix: "/etc/cni/net.d" 2022-04-02T03:08:14.200 INFO:teuthology.orchestra.run.smithi029.stdout: - pathPrefix: "/etc/kube-flannel" 2022-04-02T03:08:14.200 INFO:teuthology.orchestra.run.smithi029.stdout: - pathPrefix: "/run/flannel" 2022-04-02T03:08:14.200 INFO:teuthology.orchestra.run.smithi029.stdout: readOnlyRootFilesystem: false 2022-04-02T03:08:14.201 INFO:teuthology.orchestra.run.smithi029.stdout: # Users and groups 2022-04-02T03:08:14.201 INFO:teuthology.orchestra.run.smithi029.stdout: runAsUser: 2022-04-02T03:08:14.201 INFO:teuthology.orchestra.run.smithi029.stdout: rule: RunAsAny 2022-04-02T03:08:14.201 INFO:teuthology.orchestra.run.smithi029.stdout: supplementalGroups: 2022-04-02T03:08:14.202 INFO:teuthology.orchestra.run.smithi029.stdout: rule: RunAsAny 2022-04-02T03:08:14.202 INFO:teuthology.orchestra.run.smithi029.stdout: fsGroup: 2022-04-02T03:08:14.202 INFO:teuthology.orchestra.run.smithi029.stdout: rule: RunAsAny 2022-04-02T03:08:14.202 INFO:teuthology.orchestra.run.smithi029.stdout: # Privilege Escalation 2022-04-02T03:08:14.203 INFO:teuthology.orchestra.run.smithi029.stdout: allowPrivilegeEscalation: false 2022-04-02T03:08:14.203 INFO:teuthology.orchestra.run.smithi029.stdout: defaultAllowPrivilegeEscalation: false 2022-04-02T03:08:14.203 INFO:teuthology.orchestra.run.smithi029.stdout: # Capabilities 2022-04-02T03:08:14.203 INFO:teuthology.orchestra.run.smithi029.stdout: allowedCapabilities: ['NET_ADMIN', 'NET_RAW'] 2022-04-02T03:08:14.204 INFO:teuthology.orchestra.run.smithi029.stdout: defaultAddCapabilities: [] 2022-04-02T03:08:14.204 INFO:teuthology.orchestra.run.smithi029.stdout: requiredDropCapabilities: [] 2022-04-02T03:08:14.204 INFO:teuthology.orchestra.run.smithi029.stdout: # Host namespaces 2022-04-02T03:08:14.204 INFO:teuthology.orchestra.run.smithi029.stdout: hostPID: false 2022-04-02T03:08:14.205 INFO:teuthology.orchestra.run.smithi029.stdout: hostIPC: false 2022-04-02T03:08:14.205 INFO:teuthology.orchestra.run.smithi029.stdout: hostNetwork: true 2022-04-02T03:08:14.205 INFO:teuthology.orchestra.run.smithi029.stdout: hostPorts: 2022-04-02T03:08:14.205 INFO:teuthology.orchestra.run.smithi029.stdout: - min: 0 2022-04-02T03:08:14.205 INFO:teuthology.orchestra.run.smithi029.stdout: max: 65535 2022-04-02T03:08:14.206 INFO:teuthology.orchestra.run.smithi029.stdout: # SELinux 2022-04-02T03:08:14.206 INFO:teuthology.orchestra.run.smithi029.stdout: seLinux: 2022-04-02T03:08:14.206 INFO:teuthology.orchestra.run.smithi029.stdout: # SELinux is unused in CaaSP 2022-04-02T03:08:14.206 INFO:teuthology.orchestra.run.smithi029.stdout: rule: 'RunAsAny' 2022-04-02T03:08:14.207 INFO:teuthology.orchestra.run.smithi029.stdout:--- 2022-04-02T03:08:14.207 INFO:teuthology.orchestra.run.smithi029.stdout:kind: ClusterRole 2022-04-02T03:08:14.207 INFO:teuthology.orchestra.run.smithi029.stdout:apiVersion: rbac.authorization.k8s.io/v1 2022-04-02T03:08:14.207 INFO:teuthology.orchestra.run.smithi029.stdout:metadata: 2022-04-02T03:08:14.208 INFO:teuthology.orchestra.run.smithi029.stdout: name: flannel 2022-04-02T03:08:14.208 INFO:teuthology.orchestra.run.smithi029.stdout:rules: 2022-04-02T03:08:14.208 INFO:teuthology.orchestra.run.smithi029.stdout:- apiGroups: ['extensions'] 2022-04-02T03:08:14.208 INFO:teuthology.orchestra.run.smithi029.stdout: resources: ['podsecuritypolicies'] 2022-04-02T03:08:14.209 INFO:teuthology.orchestra.run.smithi029.stdout: verbs: ['use'] 2022-04-02T03:08:14.209 INFO:teuthology.orchestra.run.smithi029.stdout: resourceNames: ['psp.flannel.unprivileged'] 2022-04-02T03:08:14.209 INFO:teuthology.orchestra.run.smithi029.stdout:- apiGroups: 2022-04-02T03:08:14.209 INFO:teuthology.orchestra.run.smithi029.stdout: - "" 2022-04-02T03:08:14.210 INFO:teuthology.orchestra.run.smithi029.stdout: resources: 2022-04-02T03:08:14.210 INFO:teuthology.orchestra.run.smithi029.stdout: - pods 2022-04-02T03:08:14.210 INFO:teuthology.orchestra.run.smithi029.stdout: verbs: 2022-04-02T03:08:14.210 INFO:teuthology.orchestra.run.smithi029.stdout: - get 2022-04-02T03:08:14.211 INFO:teuthology.orchestra.run.smithi029.stdout:- apiGroups: 2022-04-02T03:08:14.211 INFO:teuthology.orchestra.run.smithi029.stdout: - "" 2022-04-02T03:08:14.211 INFO:teuthology.orchestra.run.smithi029.stdout: resources: 2022-04-02T03:08:14.211 INFO:teuthology.orchestra.run.smithi029.stdout: - nodes 2022-04-02T03:08:14.211 INFO:teuthology.orchestra.run.smithi029.stdout: verbs: 2022-04-02T03:08:14.212 INFO:teuthology.orchestra.run.smithi029.stdout: - list 2022-04-02T03:08:14.212 INFO:teuthology.orchestra.run.smithi029.stdout: - watch 2022-04-02T03:08:14.212 INFO:teuthology.orchestra.run.smithi029.stdout:- apiGroups: 2022-04-02T03:08:14.212 INFO:teuthology.orchestra.run.smithi029.stdout: - "" 2022-04-02T03:08:14.213 INFO:teuthology.orchestra.run.smithi029.stdout: resources: 2022-04-02T03:08:14.213 INFO:teuthology.orchestra.run.smithi029.stdout: - nodes/status 2022-04-02T03:08:14.213 INFO:teuthology.orchestra.run.smithi029.stdout: verbs: 2022-04-02T03:08:14.213 INFO:teuthology.orchestra.run.smithi029.stdout: - patch 2022-04-02T03:08:14.214 INFO:teuthology.orchestra.run.smithi029.stdout:--- 2022-04-02T03:08:14.214 INFO:teuthology.orchestra.run.smithi029.stdout:kind: ClusterRoleBinding 2022-04-02T03:08:14.214 INFO:teuthology.orchestra.run.smithi029.stdout:apiVersion: rbac.authorization.k8s.io/v1 2022-04-02T03:08:14.214 INFO:teuthology.orchestra.run.smithi029.stdout:metadata: 2022-04-02T03:08:14.215 INFO:teuthology.orchestra.run.smithi029.stdout: name: flannel 2022-04-02T03:08:14.215 INFO:teuthology.orchestra.run.smithi029.stdout:roleRef: 2022-04-02T03:08:14.215 INFO:teuthology.orchestra.run.smithi029.stdout: apiGroup: rbac.authorization.k8s.io 2022-04-02T03:08:14.215 INFO:teuthology.orchestra.run.smithi029.stdout: kind: ClusterRole 2022-04-02T03:08:14.215 INFO:teuthology.orchestra.run.smithi029.stdout: name: flannel 2022-04-02T03:08:14.216 INFO:teuthology.orchestra.run.smithi029.stdout:subjects: 2022-04-02T03:08:14.216 INFO:teuthology.orchestra.run.smithi029.stdout:- kind: ServiceAccount 2022-04-02T03:08:14.216 INFO:teuthology.orchestra.run.smithi029.stdout: name: flannel 2022-04-02T03:08:14.216 INFO:teuthology.orchestra.run.smithi029.stdout: namespace: kube-system 2022-04-02T03:08:14.216 INFO:teuthology.orchestra.run.smithi029.stdout:--- 2022-04-02T03:08:14.217 INFO:teuthology.orchestra.run.smithi029.stdout:apiVersion: v1 2022-04-02T03:08:14.217 INFO:teuthology.orchestra.run.smithi029.stdout:kind: ServiceAccount 2022-04-02T03:08:14.217 INFO:teuthology.orchestra.run.smithi029.stdout:metadata: 2022-04-02T03:08:14.218 INFO:teuthology.orchestra.run.smithi029.stdout: name: flannel 2022-04-02T03:08:14.218 INFO:teuthology.orchestra.run.smithi029.stdout: namespace: kube-system 2022-04-02T03:08:14.218 INFO:teuthology.orchestra.run.smithi029.stdout:--- 2022-04-02T03:08:14.219 INFO:teuthology.orchestra.run.smithi029.stdout:kind: ConfigMap 2022-04-02T03:08:14.219 INFO:teuthology.orchestra.run.smithi029.stdout:apiVersion: v1 2022-04-02T03:08:14.219 INFO:teuthology.orchestra.run.smithi029.stdout:metadata: 2022-04-02T03:08:14.219 INFO:teuthology.orchestra.run.smithi029.stdout: name: kube-flannel-cfg 2022-04-02T03:08:14.219 INFO:teuthology.orchestra.run.smithi029.stdout: namespace: kube-system 2022-04-02T03:08:14.220 INFO:teuthology.orchestra.run.smithi029.stdout: labels: 2022-04-02T03:08:14.220 INFO:teuthology.orchestra.run.smithi029.stdout: tier: node 2022-04-02T03:08:14.220 INFO:teuthology.orchestra.run.smithi029.stdout: app: flannel 2022-04-02T03:08:14.220 INFO:teuthology.orchestra.run.smithi029.stdout:data: 2022-04-02T03:08:14.220 INFO:teuthology.orchestra.run.smithi029.stdout: cni-conf.json: | 2022-04-02T03:08:14.221 INFO:teuthology.orchestra.run.smithi029.stdout: { 2022-04-02T03:08:14.221 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "cbr0", 2022-04-02T03:08:14.221 INFO:teuthology.orchestra.run.smithi029.stdout: "cniVersion": "0.3.1", 2022-04-02T03:08:14.221 INFO:teuthology.orchestra.run.smithi029.stdout: "plugins": [ 2022-04-02T03:08:14.222 INFO:teuthology.orchestra.run.smithi029.stdout: { 2022-04-02T03:08:14.222 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "flannel", 2022-04-02T03:08:14.222 INFO:teuthology.orchestra.run.smithi029.stdout: "delegate": { 2022-04-02T03:08:14.222 INFO:teuthology.orchestra.run.smithi029.stdout: "hairpinMode": true, 2022-04-02T03:08:14.223 INFO:teuthology.orchestra.run.smithi029.stdout: "isDefaultGateway": true 2022-04-02T03:08:14.223 INFO:teuthology.orchestra.run.smithi029.stdout: } 2022-04-02T03:08:14.223 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2022-04-02T03:08:14.223 INFO:teuthology.orchestra.run.smithi029.stdout: { 2022-04-02T03:08:14.224 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "portmap", 2022-04-02T03:08:14.224 INFO:teuthology.orchestra.run.smithi029.stdout: "capabilities": { 2022-04-02T03:08:14.224 INFO:teuthology.orchestra.run.smithi029.stdout: "portMappings": true 2022-04-02T03:08:14.224 INFO:teuthology.orchestra.run.smithi029.stdout: } 2022-04-02T03:08:14.225 INFO:teuthology.orchestra.run.smithi029.stdout: } 2022-04-02T03:08:14.225 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2022-04-02T03:08:14.225 INFO:teuthology.orchestra.run.smithi029.stdout: } 2022-04-02T03:08:14.225 INFO:teuthology.orchestra.run.smithi029.stdout: net-conf.json: | 2022-04-02T03:08:14.226 INFO:teuthology.orchestra.run.smithi029.stdout: { 2022-04-02T03:08:14.226 INFO:teuthology.orchestra.run.smithi029.stdout: "Network": "10.244.0.0/16", 2022-04-02T03:08:14.226 INFO:teuthology.orchestra.run.smithi029.stdout: "Backend": { 2022-04-02T03:08:14.227 INFO:teuthology.orchestra.run.smithi029.stdout: "Type": "vxlan" 2022-04-02T03:08:14.227 INFO:teuthology.orchestra.run.smithi029.stdout: } 2022-04-02T03:08:14.227 INFO:teuthology.orchestra.run.smithi029.stdout: } 2022-04-02T03:08:14.227 INFO:teuthology.orchestra.run.smithi029.stdout:--- 2022-04-02T03:08:14.227 INFO:teuthology.orchestra.run.smithi029.stdout:apiVersion: apps/v1 2022-04-02T03:08:14.228 INFO:teuthology.orchestra.run.smithi029.stdout:kind: DaemonSet 2022-04-02T03:08:14.228 INFO:teuthology.orchestra.run.smithi029.stdout:metadata: 2022-04-02T03:08:14.228 INFO:teuthology.orchestra.run.smithi029.stdout: name: kube-flannel-ds 2022-04-02T03:08:14.228 INFO:teuthology.orchestra.run.smithi029.stdout: namespace: kube-system 2022-04-02T03:08:14.229 INFO:teuthology.orchestra.run.smithi029.stdout: labels: 2022-04-02T03:08:14.229 INFO:teuthology.orchestra.run.smithi029.stdout: tier: node 2022-04-02T03:08:14.229 INFO:teuthology.orchestra.run.smithi029.stdout: app: flannel 2022-04-02T03:08:14.230 INFO:teuthology.orchestra.run.smithi029.stdout:spec: 2022-04-02T03:08:14.230 INFO:teuthology.orchestra.run.smithi029.stdout: selector: 2022-04-02T03:08:14.230 INFO:teuthology.orchestra.run.smithi029.stdout: matchLabels: 2022-04-02T03:08:14.230 INFO:teuthology.orchestra.run.smithi029.stdout: app: flannel 2022-04-02T03:08:14.230 INFO:teuthology.orchestra.run.smithi029.stdout: template: 2022-04-02T03:08:14.231 INFO:teuthology.orchestra.run.smithi029.stdout: metadata: 2022-04-02T03:08:14.231 INFO:teuthology.orchestra.run.smithi029.stdout: labels: 2022-04-02T03:08:14.231 INFO:teuthology.orchestra.run.smithi029.stdout: tier: node 2022-04-02T03:08:14.231 INFO:teuthology.orchestra.run.smithi029.stdout: app: flannel 2022-04-02T03:08:14.232 INFO:teuthology.orchestra.run.smithi029.stdout: spec: 2022-04-02T03:08:14.232 INFO:teuthology.orchestra.run.smithi029.stdout: affinity: 2022-04-02T03:08:14.232 INFO:teuthology.orchestra.run.smithi029.stdout: nodeAffinity: 2022-04-02T03:08:14.232 INFO:teuthology.orchestra.run.smithi029.stdout: requiredDuringSchedulingIgnoredDuringExecution: 2022-04-02T03:08:14.232 INFO:teuthology.orchestra.run.smithi029.stdout: nodeSelectorTerms: 2022-04-02T03:08:14.233 INFO:teuthology.orchestra.run.smithi029.stdout: - matchExpressions: 2022-04-02T03:08:14.233 INFO:teuthology.orchestra.run.smithi029.stdout: - key: kubernetes.io/os 2022-04-02T03:08:14.233 INFO:teuthology.orchestra.run.smithi029.stdout: operator: In 2022-04-02T03:08:14.233 INFO:teuthology.orchestra.run.smithi029.stdout: values: 2022-04-02T03:08:14.234 INFO:teuthology.orchestra.run.smithi029.stdout: - linux 2022-04-02T03:08:14.234 INFO:teuthology.orchestra.run.smithi029.stdout: hostNetwork: true 2022-04-02T03:08:14.234 INFO:teuthology.orchestra.run.smithi029.stdout: priorityClassName: system-node-critical 2022-04-02T03:08:14.234 INFO:teuthology.orchestra.run.smithi029.stdout: tolerations: 2022-04-02T03:08:14.235 INFO:teuthology.orchestra.run.smithi029.stdout: - operator: Exists 2022-04-02T03:08:14.235 INFO:teuthology.orchestra.run.smithi029.stdout: effect: NoSchedule 2022-04-02T03:08:14.235 INFO:teuthology.orchestra.run.smithi029.stdout: serviceAccountName: flannel 2022-04-02T03:08:14.235 INFO:teuthology.orchestra.run.smithi029.stdout: initContainers: 2022-04-02T03:08:14.236 INFO:teuthology.orchestra.run.smithi029.stdout: - name: install-cni-plugin 2022-04-02T03:08:14.236 INFO:teuthology.orchestra.run.smithi029.stdout: #image: flannelcni/flannel-cni-plugin:v1.0.1 for ppc64le and mips64le (dockerhub limitations may apply) 2022-04-02T03:08:14.236 INFO:teuthology.orchestra.run.smithi029.stdout: image: rancher/mirrored-flannelcni-flannel-cni-plugin:v1.0.1 2022-04-02T03:08:14.236 INFO:teuthology.orchestra.run.smithi029.stdout: command: 2022-04-02T03:08:14.236 INFO:teuthology.orchestra.run.smithi029.stdout: - cp 2022-04-02T03:08:14.237 INFO:teuthology.orchestra.run.smithi029.stdout: args: 2022-04-02T03:08:14.237 INFO:teuthology.orchestra.run.smithi029.stdout: - -f 2022-04-02T03:08:14.237 INFO:teuthology.orchestra.run.smithi029.stdout: - /flannel 2022-04-02T03:08:14.237 INFO:teuthology.orchestra.run.smithi029.stdout: - /opt/cni/bin/flannel 2022-04-02T03:08:14.238 INFO:teuthology.orchestra.run.smithi029.stdout: volumeMounts: 2022-04-02T03:08:14.238 INFO:teuthology.orchestra.run.smithi029.stdout: - name: cni-plugin 2022-04-02T03:08:14.238 INFO:teuthology.orchestra.run.smithi029.stdout: mountPath: /opt/cni/bin 2022-04-02T03:08:14.238 INFO:teuthology.orchestra.run.smithi029.stdout: - name: install-cni 2022-04-02T03:08:14.238 INFO:teuthology.orchestra.run.smithi029.stdout: #image: flannelcni/flannel:v0.17.0 for ppc64le and mips64le (dockerhub limitations may apply) 2022-04-02T03:08:14.239 INFO:teuthology.orchestra.run.smithi029.stdout: image: rancher/mirrored-flannelcni-flannel:v0.17.0 2022-04-02T03:08:14.239 INFO:teuthology.orchestra.run.smithi029.stdout: command: 2022-04-02T03:08:14.239 INFO:teuthology.orchestra.run.smithi029.stdout: - cp 2022-04-02T03:08:14.240 INFO:teuthology.orchestra.run.smithi029.stdout: args: 2022-04-02T03:08:14.240 INFO:teuthology.orchestra.run.smithi029.stdout: - -f 2022-04-02T03:08:14.240 INFO:teuthology.orchestra.run.smithi029.stdout: - /etc/kube-flannel/cni-conf.json 2022-04-02T03:08:14.241 INFO:teuthology.orchestra.run.smithi029.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 5692 100 5692 0 0 59915 0 --:--:-- --:--:-- --:--:-- 59915 2022-04-02T03:08:14.242 INFO:teuthology.orchestra.run.smithi029.stdout: - /etc/cni/net.d/10-flannel.conflist 2022-04-02T03:08:14.242 INFO:teuthology.orchestra.run.smithi029.stdout: volumeMounts: 2022-04-02T03:08:14.243 INFO:teuthology.orchestra.run.smithi029.stdout: - name: cni 2022-04-02T03:08:14.243 INFO:teuthology.orchestra.run.smithi029.stdout: mountPath: /etc/cni/net.d 2022-04-02T03:08:14.243 INFO:teuthology.orchestra.run.smithi029.stdout: - name: flannel-cfg 2022-04-02T03:08:14.243 INFO:teuthology.orchestra.run.smithi029.stdout: mountPath: /etc/kube-flannel/ 2022-04-02T03:08:14.244 INFO:teuthology.orchestra.run.smithi029.stdout: containers: 2022-04-02T03:08:14.244 INFO:teuthology.orchestra.run.smithi029.stdout: - name: kube-flannel 2022-04-02T03:08:14.244 INFO:teuthology.orchestra.run.smithi029.stdout: #image: flannelcni/flannel:v0.17.0 for ppc64le and mips64le (dockerhub limitations may apply) 2022-04-02T03:08:14.244 INFO:teuthology.orchestra.run.smithi029.stdout: image: rancher/mirrored-flannelcni-flannel:v0.17.0 2022-04-02T03:08:14.245 INFO:teuthology.orchestra.run.smithi029.stdout: command: 2022-04-02T03:08:14.245 INFO:teuthology.orchestra.run.smithi029.stdout: - /opt/bin/flanneld 2022-04-02T03:08:14.245 INFO:teuthology.orchestra.run.smithi029.stdout: args: 2022-04-02T03:08:14.245 INFO:teuthology.orchestra.run.smithi029.stdout: - --ip-masq 2022-04-02T03:08:14.246 INFO:teuthology.orchestra.run.smithi029.stdout: - --kube-subnet-mgr 2022-04-02T03:08:14.246 INFO:teuthology.orchestra.run.smithi029.stdout: resources: 2022-04-02T03:08:14.246 INFO:teuthology.orchestra.run.smithi029.stdout: requests: 2022-04-02T03:08:14.247 INFO:teuthology.orchestra.run.smithi029.stdout: cpu: "100m" 2022-04-02T03:08:14.247 INFO:teuthology.orchestra.run.smithi029.stdout: memory: "50Mi" 2022-04-02T03:08:14.247 INFO:teuthology.orchestra.run.smithi029.stdout: limits: 2022-04-02T03:08:14.247 INFO:teuthology.orchestra.run.smithi029.stdout: cpu: "100m" 2022-04-02T03:08:14.247 INFO:teuthology.orchestra.run.smithi029.stdout: memory: "50Mi" 2022-04-02T03:08:14.248 INFO:teuthology.orchestra.run.smithi029.stdout: securityContext: 2022-04-02T03:08:14.248 INFO:teuthology.orchestra.run.smithi029.stdout: privileged: false 2022-04-02T03:08:14.248 INFO:teuthology.orchestra.run.smithi029.stdout: capabilities: 2022-04-02T03:08:14.248 INFO:teuthology.orchestra.run.smithi029.stdout: add: ["NET_ADMIN", "NET_RAW"] 2022-04-02T03:08:14.249 INFO:teuthology.orchestra.run.smithi029.stdout: env: 2022-04-02T03:08:14.249 INFO:teuthology.orchestra.run.smithi029.stdout: - name: POD_NAME 2022-04-02T03:08:14.249 INFO:teuthology.orchestra.run.smithi029.stdout: valueFrom: 2022-04-02T03:08:14.249 INFO:teuthology.orchestra.run.smithi029.stdout: fieldRef: 2022-04-02T03:08:14.249 INFO:teuthology.orchestra.run.smithi029.stdout: fieldPath: metadata.name 2022-04-02T03:08:14.250 INFO:teuthology.orchestra.run.smithi029.stdout: - name: POD_NAMESPACE 2022-04-02T03:08:14.250 INFO:teuthology.orchestra.run.smithi029.stdout: valueFrom: 2022-04-02T03:08:14.251 INFO:teuthology.orchestra.run.smithi029.stdout: fieldRef: 2022-04-02T03:08:14.251 INFO:teuthology.orchestra.run.smithi029.stdout: fieldPath: metadata.namespace 2022-04-02T03:08:14.251 INFO:teuthology.orchestra.run.smithi029.stdout: volumeMounts: 2022-04-02T03:08:14.251 INFO:teuthology.orchestra.run.smithi029.stdout: - name: run 2022-04-02T03:08:14.252 INFO:teuthology.orchestra.run.smithi029.stdout: mountPath: /run/flannel 2022-04-02T03:08:14.252 INFO:teuthology.orchestra.run.smithi029.stdout: - name: flannel-cfg 2022-04-02T03:08:14.252 INFO:teuthology.orchestra.run.smithi029.stdout: mountPath: /etc/kube-flannel/ 2022-04-02T03:08:14.252 INFO:teuthology.orchestra.run.smithi029.stdout: - name: xtables-lock 2022-04-02T03:08:14.253 INFO:teuthology.orchestra.run.smithi029.stdout: mountPath: /run/xtables.lock 2022-04-02T03:08:14.253 INFO:teuthology.orchestra.run.smithi029.stdout: volumes: 2022-04-02T03:08:14.253 INFO:teuthology.orchestra.run.smithi029.stdout: - name: run 2022-04-02T03:08:14.253 INFO:teuthology.orchestra.run.smithi029.stdout: hostPath: 2022-04-02T03:08:14.253 INFO:teuthology.orchestra.run.smithi029.stdout: path: /run/flannel 2022-04-02T03:08:14.254 INFO:teuthology.orchestra.run.smithi029.stdout: - name: cni-plugin 2022-04-02T03:08:14.254 INFO:teuthology.orchestra.run.smithi029.stdout: hostPath: 2022-04-02T03:08:14.254 INFO:teuthology.orchestra.run.smithi029.stdout: path: /opt/cni/bin 2022-04-02T03:08:14.255 INFO:teuthology.orchestra.run.smithi029.stdout: - name: cni 2022-04-02T03:08:14.255 INFO:teuthology.orchestra.run.smithi029.stdout: hostPath: 2022-04-02T03:08:14.255 INFO:teuthology.orchestra.run.smithi029.stdout: path: /etc/cni/net.d 2022-04-02T03:08:14.256 INFO:teuthology.orchestra.run.smithi029.stdout: - name: flannel-cfg 2022-04-02T03:08:14.256 INFO:teuthology.orchestra.run.smithi029.stdout: configMap: 2022-04-02T03:08:14.256 INFO:teuthology.orchestra.run.smithi029.stdout: name: kube-flannel-cfg 2022-04-02T03:08:14.256 INFO:teuthology.orchestra.run.smithi029.stdout: - name: xtables-lock 2022-04-02T03:08:14.256 INFO:teuthology.orchestra.run.smithi029.stdout: hostPath: 2022-04-02T03:08:14.257 INFO:teuthology.orchestra.run.smithi029.stdout: path: /run/xtables.lock 2022-04-02T03:08:14.257 INFO:teuthology.orchestra.run.smithi029.stdout: type: FileOrCreate 2022-04-02T03:08:14.287 INFO:tasks.kubeadm:Updating {'kind': 'ConfigMap', 'apiVersion': 'v1', 'metadata': {'name': 'kube-flannel-cfg', 'namespace': 'kube-system', 'labels': {'tier': 'node', 'app': 'flannel'}}, 'data': {'cni-conf.json': '{\n "name": "cbr0",\n "cniVersion": "0.3.1",\n "plugins": [\n {\n "type": "flannel",\n "delegate": {\n "hairpinMode": true,\n "isDefaultGateway": true\n }\n },\n {\n "type": "portmap",\n "capabilities": {\n "portMappings": true\n }\n }\n ]\n}\n', 'net-conf.json': '{\n "Network": "10.244.0.0/16",\n "Backend": {\n "Type": "vxlan"\n }\n}\n'}} 2022-04-02T03:08:14.288 INFO:tasks.kubeadm:Now {'kind': 'ConfigMap', 'apiVersion': 'v1', 'metadata': {'name': 'kube-flannel-cfg', 'namespace': 'kube-system', 'labels': {'tier': 'node', 'app': 'flannel'}}, 'data': {'cni-conf.json': '{\n "name": "cbr0",\n "cniVersion": "0.3.1",\n "plugins": [\n {\n "type": "flannel",\n "delegate": {\n "hairpinMode": true,\n "isDefaultGateway": true\n }\n },\n {\n "type": "portmap",\n "capabilities": {\n "portMappings": true\n }\n }\n ]\n}\n', 'net-conf.json': '{\n "Network": "10.248.224.0/21",\n "Backend": {\n "Type": "vxlan"\n }\n}\n'}} 2022-04-02T03:08:14.302 DEBUG:tasks.kubeadm:Flannel: apiVersion: policy/v1beta1 kind: PodSecurityPolicy metadata: annotations: apparmor.security.beta.kubernetes.io/allowedProfileNames: runtime/default apparmor.security.beta.kubernetes.io/defaultProfileName: runtime/default seccomp.security.alpha.kubernetes.io/allowedProfileNames: docker/default seccomp.security.alpha.kubernetes.io/defaultProfileName: docker/default name: psp.flannel.unprivileged spec: allowPrivilegeEscalation: false allowedCapabilities: - NET_ADMIN - NET_RAW allowedHostPaths: - pathPrefix: /etc/cni/net.d - pathPrefix: /etc/kube-flannel - pathPrefix: /run/flannel defaultAddCapabilities: [] defaultAllowPrivilegeEscalation: false fsGroup: rule: RunAsAny hostIPC: false hostNetwork: true hostPID: false hostPorts: - max: 65535 min: 0 privileged: false readOnlyRootFilesystem: false requiredDropCapabilities: [] runAsUser: rule: RunAsAny seLinux: rule: RunAsAny supplementalGroups: rule: RunAsAny volumes: - configMap - secret - emptyDir - hostPath --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: flannel rules: - apiGroups: - extensions resourceNames: - psp.flannel.unprivileged resources: - podsecuritypolicies verbs: - use - apiGroups: - '' resources: - pods verbs: - get - apiGroups: - '' resources: - nodes verbs: - list - watch - apiGroups: - '' resources: - nodes/status verbs: - patch --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: flannel roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: flannel subjects: - kind: ServiceAccount name: flannel namespace: kube-system --- apiVersion: v1 kind: ServiceAccount metadata: name: flannel namespace: kube-system --- apiVersion: v1 data: cni-conf.json: "{\n \"name\": \"cbr0\",\n \"cniVersion\": \"0.3.1\",\n \"plugins\"\ : [\n {\n \"type\": \"flannel\",\n \"delegate\": {\n \"hairpinMode\"\ : true,\n \"isDefaultGateway\": true\n }\n },\n {\n \"\ type\": \"portmap\",\n \"capabilities\": {\n \"portMappings\": true\n\ \ }\n }\n ]\n}\n" net-conf.json: "{\n \"Network\": \"10.248.224.0/21\",\n \"Backend\": {\n \"\ Type\": \"vxlan\"\n }\n}\n" kind: ConfigMap metadata: labels: app: flannel tier: node name: kube-flannel-cfg namespace: kube-system --- apiVersion: apps/v1 kind: DaemonSet metadata: labels: app: flannel tier: node name: kube-flannel-ds namespace: kube-system spec: selector: matchLabels: app: flannel template: metadata: labels: app: flannel tier: node spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/os operator: In values: - linux containers: - args: - --ip-masq - --kube-subnet-mgr command: - /opt/bin/flanneld env: - name: POD_NAME valueFrom: fieldRef: fieldPath: metadata.name - name: POD_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace image: rancher/mirrored-flannelcni-flannel:v0.17.0 name: kube-flannel resources: limits: cpu: 100m memory: 50Mi requests: cpu: 100m memory: 50Mi securityContext: capabilities: add: - NET_ADMIN - NET_RAW privileged: false volumeMounts: - mountPath: /run/flannel name: run - mountPath: /etc/kube-flannel/ name: flannel-cfg - mountPath: /run/xtables.lock name: xtables-lock hostNetwork: true initContainers: - args: - -f - /flannel - /opt/cni/bin/flannel command: - cp image: rancher/mirrored-flannelcni-flannel-cni-plugin:v1.0.1 name: install-cni-plugin volumeMounts: - mountPath: /opt/cni/bin name: cni-plugin - args: - -f - /etc/kube-flannel/cni-conf.json - /etc/cni/net.d/10-flannel.conflist command: - cp image: rancher/mirrored-flannelcni-flannel:v0.17.0 name: install-cni volumeMounts: - mountPath: /etc/cni/net.d name: cni - mountPath: /etc/kube-flannel/ name: flannel-cfg priorityClassName: system-node-critical serviceAccountName: flannel tolerations: - effect: NoSchedule operator: Exists volumes: - hostPath: path: /run/flannel name: run - hostPath: path: /opt/cni/bin name: cni-plugin - hostPath: path: /etc/cni/net.d name: cni - configMap: name: kube-flannel-cfg name: flannel-cfg - hostPath: path: /run/xtables.lock type: FileOrCreate name: xtables-lock 2022-04-02T03:08:14.303 DEBUG:teuthology.orchestra.run.smithi029:> kubectl apply -f - 2022-04-02T03:08:16.842 INFO:teuthology.orchestra.run.smithi029.stderr:Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+ 2022-04-02T03:08:16.847 INFO:teuthology.orchestra.run.smithi029.stdout:podsecuritypolicy.policy/psp.flannel.unprivileged created 2022-04-02T03:08:16.901 INFO:teuthology.orchestra.run.smithi029.stdout:clusterrole.rbac.authorization.k8s.io/flannel created 2022-04-02T03:08:16.910 INFO:teuthology.orchestra.run.smithi029.stdout:clusterrolebinding.rbac.authorization.k8s.io/flannel created 2022-04-02T03:08:16.920 INFO:teuthology.orchestra.run.smithi029.stdout:serviceaccount/flannel created 2022-04-02T03:08:16.948 INFO:teuthology.orchestra.run.smithi029.stdout:configmap/kube-flannel-cfg created 2022-04-02T03:08:16.992 INFO:teuthology.orchestra.run.smithi029.stdout:daemonset.apps/kube-flannel-ds created 2022-04-02T03:08:16.996 INFO:tasks.kubeadm:Scanning for scratch devices 2022-04-02T03:08:16.996 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2022-04-02T03:08:16.996 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/scratch_devs of=/dev/stdout 2022-04-02T03:08:17.002 INFO:tasks.kubeadm:Scratch devices on smithi029: ['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-04-02T03:08:17.003 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 2022-04-02T03:08:17.023 INFO:teuthology.orchestra.run.smithi029.stderr:10+0 records in 2022-04-02T03:08:17.024 INFO:teuthology.orchestra.run.smithi029.stderr:10+0 records out 2022-04-02T03:08:17.024 INFO:teuthology.orchestra.run.smithi029.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0110897 s, 946 MB/s 2022-04-02T03:08:17.026 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 2022-04-02T03:08:17.094 INFO:teuthology.orchestra.run.smithi029.stderr:10+0 records in 2022-04-02T03:08:17.095 INFO:teuthology.orchestra.run.smithi029.stderr:10+0 records out 2022-04-02T03:08:17.095 INFO:teuthology.orchestra.run.smithi029.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0135161 s, 776 MB/s 2022-04-02T03:08:17.096 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 2022-04-02T03:08:17.115 INFO:teuthology.orchestra.run.smithi029.stderr:10+0 records in 2022-04-02T03:08:17.116 INFO:teuthology.orchestra.run.smithi029.stderr:10+0 records out 2022-04-02T03:08:17.117 INFO:teuthology.orchestra.run.smithi029.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0110645 s, 948 MB/s 2022-04-02T03:08:17.118 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 2022-04-02T03:08:17.142 INFO:teuthology.orchestra.run.smithi029.stderr:10+0 records in 2022-04-02T03:08:17.143 INFO:teuthology.orchestra.run.smithi029.stderr:10+0 records out 2022-04-02T03:08:17.143 INFO:teuthology.orchestra.run.smithi029.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0113532 s, 924 MB/s 2022-04-02T03:08:17.145 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2022-04-02T03:08:17.145 DEBUG:teuthology.orchestra.run.smithi104:> dd if=/scratch_devs of=/dev/stdout 2022-04-02T03:08:17.154 INFO:tasks.kubeadm:Scratch devices on smithi104: ['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-04-02T03:08:17.154 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 2022-04-02T03:08:17.184 INFO:teuthology.orchestra.run.smithi104.stderr:10+0 records in 2022-04-02T03:08:17.184 INFO:teuthology.orchestra.run.smithi104.stderr:10+0 records out 2022-04-02T03:08:17.185 INFO:teuthology.orchestra.run.smithi104.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0177002 s, 592 MB/s 2022-04-02T03:08:17.186 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 2022-04-02T03:08:17.215 INFO:teuthology.orchestra.run.smithi104.stderr:10+0 records in 2022-04-02T03:08:17.216 INFO:teuthology.orchestra.run.smithi104.stderr:10+0 records out 2022-04-02T03:08:17.216 INFO:teuthology.orchestra.run.smithi104.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0168339 s, 623 MB/s 2022-04-02T03:08:17.219 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 2022-04-02T03:08:17.293 INFO:teuthology.orchestra.run.smithi104.stderr:10+0 records in 2022-04-02T03:08:17.293 INFO:teuthology.orchestra.run.smithi104.stderr:10+0 records out 2022-04-02T03:08:17.294 INFO:teuthology.orchestra.run.smithi104.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.01694 s, 619 MB/s 2022-04-02T03:08:17.295 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 2022-04-02T03:08:17.321 INFO:teuthology.orchestra.run.smithi104.stderr:10+0 records in 2022-04-02T03:08:17.322 INFO:teuthology.orchestra.run.smithi104.stderr:10+0 records out 2022-04-02T03:08:17.322 INFO:teuthology.orchestra.run.smithi104.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0163302 s, 642 MB/s 2022-04-02T03:08:17.323 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2022-04-02T03:08:17.323 DEBUG:teuthology.orchestra.run.smithi142:> dd if=/scratch_devs of=/dev/stdout 2022-04-02T03:08:17.331 INFO:tasks.kubeadm:Scratch devices on smithi142: ['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-04-02T03:08:17.332 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 2022-04-02T03:08:17.406 INFO:teuthology.orchestra.run.smithi142.stderr:10+0 records in 2022-04-02T03:08:17.406 INFO:teuthology.orchestra.run.smithi142.stderr:10+0 records out 2022-04-02T03:08:17.407 INFO:teuthology.orchestra.run.smithi142.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0163222 s, 642 MB/s 2022-04-02T03:08:17.408 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 2022-04-02T03:08:17.435 INFO:teuthology.orchestra.run.smithi142.stderr:10+0 records in 2022-04-02T03:08:17.435 INFO:teuthology.orchestra.run.smithi142.stderr:10+0 records out 2022-04-02T03:08:17.435 INFO:teuthology.orchestra.run.smithi142.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0149405 s, 702 MB/s 2022-04-02T03:08:17.437 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 2022-04-02T03:08:17.464 INFO:teuthology.orchestra.run.smithi142.stderr:10+0 records in 2022-04-02T03:08:17.465 INFO:teuthology.orchestra.run.smithi142.stderr:10+0 records out 2022-04-02T03:08:17.465 INFO:teuthology.orchestra.run.smithi142.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0163874 s, 640 MB/s 2022-04-02T03:08:17.467 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 2022-04-02T03:08:17.493 INFO:teuthology.orchestra.run.smithi142.stderr:10+0 records in 2022-04-02T03:08:17.493 INFO:teuthology.orchestra.run.smithi142.stderr:10+0 records out 2022-04-02T03:08:17.494 INFO:teuthology.orchestra.run.smithi142.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0143021 s, 733 MB/s 2022-04-02T03:08:17.529 INFO:tasks.kubeadm:Creating PVs + StorageClass 2022-04-02T03:08:17.530 DEBUG:tasks.kubeadm:apiVersion: v1 kind: PersistentVolume metadata: name: smithi029-lv-1 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_1 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi029 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi029-lv-2 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_2 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi029 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi029-lv-3 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_3 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi029 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi029-lv-4 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_4 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi029 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi104-lv-1 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_1 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi104 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi104-lv-2 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_2 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi104 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi104-lv-3 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_3 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi104 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi104-lv-4 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_4 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi104 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi142-lv-1 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_1 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi142 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi142-lv-2 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_2 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi142 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi142-lv-3 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_3 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi142 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi142-lv-4 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_4 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi142 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: scratch provisioner: kubernetes.io/no-provisioner volumeBindingMode: WaitForFirstConsumer 2022-04-02T03:08:17.530 DEBUG:teuthology.orchestra.run.smithi029:> kubectl create -f - 2022-04-02T03:08:18.204 INFO:teuthology.orchestra.run.smithi029.stdout:persistentvolume/smithi029-lv-1 created 2022-04-02T03:08:18.250 INFO:teuthology.orchestra.run.smithi029.stdout:persistentvolume/smithi029-lv-2 created 2022-04-02T03:08:18.259 INFO:teuthology.orchestra.run.smithi029.stdout:persistentvolume/smithi029-lv-3 created 2022-04-02T03:08:18.328 INFO:teuthology.orchestra.run.smithi029.stdout:persistentvolume/smithi029-lv-4 created 2022-04-02T03:08:18.359 INFO:teuthology.orchestra.run.smithi029.stdout:persistentvolume/smithi104-lv-1 created 2022-04-02T03:08:18.473 INFO:teuthology.orchestra.run.smithi029.stdout:persistentvolume/smithi104-lv-2 created 2022-04-02T03:08:18.568 INFO:teuthology.orchestra.run.smithi029.stdout:persistentvolume/smithi104-lv-3 created 2022-04-02T03:08:18.643 INFO:teuthology.orchestra.run.smithi029.stdout:persistentvolume/smithi104-lv-4 created 2022-04-02T03:08:18.659 INFO:teuthology.orchestra.run.smithi029.stdout:persistentvolume/smithi142-lv-1 created 2022-04-02T03:08:18.667 INFO:teuthology.orchestra.run.smithi029.stdout:persistentvolume/smithi142-lv-2 created 2022-04-02T03:08:18.736 INFO:teuthology.orchestra.run.smithi029.stdout:persistentvolume/smithi142-lv-3 created 2022-04-02T03:08:18.780 INFO:teuthology.orchestra.run.smithi029.stdout:persistentvolume/smithi142-lv-4 created 2022-04-02T03:08:18.783 INFO:teuthology.orchestra.run.smithi029.stdout:storageclass.storage.k8s.io/scratch created 2022-04-02T03:08:18.787 DEBUG:teuthology.orchestra.run.smithi029:> kubectl taint node smithi029 node-role.kubernetes.io/master- || true 2022-04-02T03:08:18.872 INFO:teuthology.orchestra.run.smithi029.stdout:node/smithi029 untainted 2022-04-02T03:08:18.875 INFO:tasks.kubeadm:Kubeadm complete, yielding 2022-04-02T03:08:18.875 INFO:teuthology.run_tasks:Running task nvme_loop... 2022-04-02T03:08:18.897 INFO:tasks.nvme_loop:Setting up nvme_loop on scratch devices... 2022-04-02T03:08:18.899 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2022-04-02T03:08:18.899 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/scratch_devs of=/dev/stdout 2022-04-02T03:08:18.908 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-04-02T03:08:18.909 DEBUG:teuthology.orchestra.run.smithi029:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2022-04-02T03:08:18.962 INFO:teuthology.orchestra.run.smithi029.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2022-04-02T03:08:18.963 INFO:teuthology.orchestra.run.smithi029.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-04-02T03:08:18.963 INFO:teuthology.orchestra.run.smithi029.stdout:Device: 5h/5d Inode: 501 Links: 1 2022-04-02T03:08:18.964 INFO:teuthology.orchestra.run.smithi029.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-04-02T03:08:18.964 INFO:teuthology.orchestra.run.smithi029.stdout:Access: 2022-04-02 03:08:17.020095042 +0000 2022-04-02T03:08:18.965 INFO:teuthology.orchestra.run.smithi029.stdout:Modify: 2022-04-02 03:08:17.020095042 +0000 2022-04-02T03:08:18.965 INFO:teuthology.orchestra.run.smithi029.stdout:Change: 2022-04-02 03:08:17.020095042 +0000 2022-04-02T03:08:18.965 INFO:teuthology.orchestra.run.smithi029.stdout: Birth: - 2022-04-02T03:08:18.971 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records in 2022-04-02T03:08:18.972 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records out 2022-04-02T03:08:18.972 INFO:teuthology.orchestra.run.smithi029.stderr:512 bytes copied, 0.000329178 s, 1.6 MB/s 2022-04-02T03:08:18.977 DEBUG:teuthology.orchestra.run.smithi029:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2022-04-02T03:08:18.985 INFO:teuthology.orchestra.run.smithi029.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2022-04-02T03:08:18.986 INFO:teuthology.orchestra.run.smithi029.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-04-02T03:08:18.986 INFO:teuthology.orchestra.run.smithi029.stdout:Device: 5h/5d Inode: 507 Links: 1 2022-04-02T03:08:18.986 INFO:teuthology.orchestra.run.smithi029.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-04-02T03:08:18.986 INFO:teuthology.orchestra.run.smithi029.stdout:Access: 2022-04-02 03:08:17.092095044 +0000 2022-04-02T03:08:18.987 INFO:teuthology.orchestra.run.smithi029.stdout:Modify: 2022-04-02 03:08:17.092095044 +0000 2022-04-02T03:08:18.987 INFO:teuthology.orchestra.run.smithi029.stdout:Change: 2022-04-02 03:08:17.092095044 +0000 2022-04-02T03:08:18.987 INFO:teuthology.orchestra.run.smithi029.stdout: Birth: - 2022-04-02T03:08:18.994 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records in 2022-04-02T03:08:18.994 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records out 2022-04-02T03:08:18.994 INFO:teuthology.orchestra.run.smithi029.stderr:512 bytes copied, 0.000320697 s, 1.6 MB/s 2022-04-02T03:08:18.997 DEBUG:teuthology.orchestra.run.smithi029:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2022-04-02T03:08:19.002 INFO:teuthology.orchestra.run.smithi029.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2022-04-02T03:08:19.002 INFO:teuthology.orchestra.run.smithi029.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-04-02T03:08:19.002 INFO:teuthology.orchestra.run.smithi029.stdout:Device: 5h/5d Inode: 513 Links: 1 2022-04-02T03:08:19.003 INFO:teuthology.orchestra.run.smithi029.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-04-02T03:08:19.003 INFO:teuthology.orchestra.run.smithi029.stdout:Access: 2022-04-02 03:08:17.112095045 +0000 2022-04-02T03:08:19.003 INFO:teuthology.orchestra.run.smithi029.stdout:Modify: 2022-04-02 03:08:17.112095045 +0000 2022-04-02T03:08:19.003 INFO:teuthology.orchestra.run.smithi029.stdout:Change: 2022-04-02 03:08:17.112095045 +0000 2022-04-02T03:08:19.004 INFO:teuthology.orchestra.run.smithi029.stdout: Birth: - 2022-04-02T03:08:19.012 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records in 2022-04-02T03:08:19.012 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records out 2022-04-02T03:08:19.013 INFO:teuthology.orchestra.run.smithi029.stderr:512 bytes copied, 0.00030375 s, 1.7 MB/s 2022-04-02T03:08:19.016 DEBUG:teuthology.orchestra.run.smithi029:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2022-04-02T03:08:19.021 INFO:teuthology.orchestra.run.smithi029.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2022-04-02T03:08:19.021 INFO:teuthology.orchestra.run.smithi029.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-04-02T03:08:19.022 INFO:teuthology.orchestra.run.smithi029.stdout:Device: 5h/5d Inode: 520 Links: 1 2022-04-02T03:08:19.022 INFO:teuthology.orchestra.run.smithi029.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-04-02T03:08:19.022 INFO:teuthology.orchestra.run.smithi029.stdout:Access: 2022-04-02 03:08:17.144095046 +0000 2022-04-02T03:08:19.024 INFO:teuthology.orchestra.run.smithi029.stdout:Modify: 2022-04-02 03:08:17.140095046 +0000 2022-04-02T03:08:19.024 INFO:teuthology.orchestra.run.smithi029.stdout:Change: 2022-04-02 03:08:17.140095046 +0000 2022-04-02T03:08:19.024 INFO:teuthology.orchestra.run.smithi029.stdout: Birth: - 2022-04-02T03:08:19.030 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records in 2022-04-02T03:08:19.031 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records out 2022-04-02T03:08:19.031 INFO:teuthology.orchestra.run.smithi029.stderr:512 bytes copied, 0.000213034 s, 2.4 MB/s 2022-04-02T03:08:19.034 DEBUG:teuthology.orchestra.run.smithi029:> sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2022-04-02T03:08:19.206 INFO:teuthology.orchestra.run.smithi029.stdout:loop 2022-04-02T03:08:19.207 INFO:tasks.nvme_loop:Connecting nvme_loop smithi029:/dev/vg_nvme/lv_1... 2022-04-02T03:08:19.208 DEBUG:teuthology.orchestra.run.smithi029:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2022-04-02T03:08:19.226 INFO:teuthology.orchestra.run.smithi029.stdout:1 2022-04-02T03:08:19.248 INFO:teuthology.orchestra.run.smithi029.stdout:/dev/vg_nvme/lv_11 2022-04-02T03:08:19.270 INFO:tasks.nvme_loop:Connecting nvme_loop smithi029:/dev/vg_nvme/lv_2... 2022-04-02T03:08:19.270 DEBUG:teuthology.orchestra.run.smithi029:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2022-04-02T03:08:19.290 INFO:teuthology.orchestra.run.smithi029.stdout:1 2022-04-02T03:08:19.315 INFO:teuthology.orchestra.run.smithi029.stdout:/dev/vg_nvme/lv_21 2022-04-02T03:08:19.336 INFO:tasks.nvme_loop:Connecting nvme_loop smithi029:/dev/vg_nvme/lv_3... 2022-04-02T03:08:19.336 DEBUG:teuthology.orchestra.run.smithi029:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2022-04-02T03:08:19.355 INFO:teuthology.orchestra.run.smithi029.stdout:1 2022-04-02T03:08:19.382 INFO:teuthology.orchestra.run.smithi029.stdout:/dev/vg_nvme/lv_31 2022-04-02T03:08:19.402 INFO:tasks.nvme_loop:Connecting nvme_loop smithi029:/dev/vg_nvme/lv_4... 2022-04-02T03:08:19.403 DEBUG:teuthology.orchestra.run.smithi029:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2022-04-02T03:08:19.465 INFO:teuthology.orchestra.run.smithi029.stdout:1 2022-04-02T03:08:19.492 INFO:teuthology.orchestra.run.smithi029.stdout:/dev/vg_nvme/lv_41 2022-04-02T03:08:19.517 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2022-04-02T03:08:19.517 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/scratch_devs of=/dev/stdout 2022-04-02T03:08:19.524 DEBUG:teuthology.orchestra.run.smithi029:> sudo nvme list 2022-04-02T03:08:19.534 INFO:teuthology.orchestra.run.smithi029.stdout:Node SN Model Namespace Usage Format FW Rev 2022-04-02T03:08:19.534 INFO:teuthology.orchestra.run.smithi029.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2022-04-02T03:08:19.535 INFO:teuthology.orchestra.run.smithi029.stdout:/dev/nvme0n1 PHMB7512009B480DGN INTEL SSDPED1D480GA 1 480.10 GB / 480.10 GB 512 B + 0 B E2010325 2022-04-02T03:08:19.535 INFO:teuthology.orchestra.run.smithi029.stdout:/dev/nvme1n1 3f5c8f8d6756bd9c Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-04-02T03:08:19.535 INFO:teuthology.orchestra.run.smithi029.stdout:/dev/nvme2n1 e1ab1019670dd3c2 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-04-02T03:08:19.536 INFO:teuthology.orchestra.run.smithi029.stdout:/dev/nvme3n1 cb0e3ef0ccbedcf1 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-04-02T03:08:19.537 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1'] 2022-04-02T03:08:20.538 DEBUG:teuthology.orchestra.run.smithi029:> sudo nvme list 2022-04-02T03:08:20.549 INFO:teuthology.orchestra.run.smithi029.stdout:Node SN Model Namespace Usage Format FW Rev 2022-04-02T03:08:20.549 INFO:teuthology.orchestra.run.smithi029.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2022-04-02T03:08:20.549 INFO:teuthology.orchestra.run.smithi029.stdout:/dev/nvme0n1 PHMB7512009B480DGN INTEL SSDPED1D480GA 1 480.10 GB / 480.10 GB 512 B + 0 B E2010325 2022-04-02T03:08:20.550 INFO:teuthology.orchestra.run.smithi029.stdout:/dev/nvme1n1 3f5c8f8d6756bd9c Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-04-02T03:08:20.550 INFO:teuthology.orchestra.run.smithi029.stdout:/dev/nvme2n1 e1ab1019670dd3c2 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-04-02T03:08:20.550 INFO:teuthology.orchestra.run.smithi029.stdout:/dev/nvme3n1 cb0e3ef0ccbedcf1 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-04-02T03:08:20.550 INFO:teuthology.orchestra.run.smithi029.stdout:/dev/nvme4n1 a6d5a7ee4e8646f1 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-04-02T03:08:20.551 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2022-04-02T03:08:20.552 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2022-04-02T03:08:20.552 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/scratch_devs 2022-04-02T03:08:20.562 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2022-04-02T03:08:20.563 DEBUG:teuthology.orchestra.run.smithi104:> dd if=/scratch_devs of=/dev/stdout 2022-04-02T03:08:20.570 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-04-02T03:08:20.571 DEBUG:teuthology.orchestra.run.smithi104:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2022-04-02T03:08:20.621 INFO:teuthology.orchestra.run.smithi104.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2022-04-02T03:08:20.621 INFO:teuthology.orchestra.run.smithi104.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-04-02T03:08:20.622 INFO:teuthology.orchestra.run.smithi104.stdout:Device: 5h/5d Inode: 499 Links: 1 2022-04-02T03:08:20.622 INFO:teuthology.orchestra.run.smithi104.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-04-02T03:08:20.623 INFO:teuthology.orchestra.run.smithi104.stdout:Access: 2022-04-02 03:08:17.183551731 +0000 2022-04-02T03:08:20.623 INFO:teuthology.orchestra.run.smithi104.stdout:Modify: 2022-04-02 03:08:17.183551731 +0000 2022-04-02T03:08:20.624 INFO:teuthology.orchestra.run.smithi104.stdout:Change: 2022-04-02 03:08:17.183551731 +0000 2022-04-02T03:08:20.624 INFO:teuthology.orchestra.run.smithi104.stdout: Birth: - 2022-04-02T03:08:20.629 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records in 2022-04-02T03:08:20.629 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records out 2022-04-02T03:08:20.630 INFO:teuthology.orchestra.run.smithi104.stderr:512 bytes copied, 0.000561576 s, 912 kB/s 2022-04-02T03:08:20.633 DEBUG:teuthology.orchestra.run.smithi104:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2022-04-02T03:08:20.639 INFO:teuthology.orchestra.run.smithi104.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2022-04-02T03:08:20.639 INFO:teuthology.orchestra.run.smithi104.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-04-02T03:08:20.640 INFO:teuthology.orchestra.run.smithi104.stdout:Device: 5h/5d Inode: 505 Links: 1 2022-04-02T03:08:20.640 INFO:teuthology.orchestra.run.smithi104.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-04-02T03:08:20.640 INFO:teuthology.orchestra.run.smithi104.stdout:Access: 2022-04-02 03:08:17.215551730 +0000 2022-04-02T03:08:20.640 INFO:teuthology.orchestra.run.smithi104.stdout:Modify: 2022-04-02 03:08:17.215551730 +0000 2022-04-02T03:08:20.640 INFO:teuthology.orchestra.run.smithi104.stdout:Change: 2022-04-02 03:08:17.215551730 +0000 2022-04-02T03:08:20.641 INFO:teuthology.orchestra.run.smithi104.stdout: Birth: - 2022-04-02T03:08:20.647 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records in 2022-04-02T03:08:20.647 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records out 2022-04-02T03:08:20.647 INFO:teuthology.orchestra.run.smithi104.stderr:512 bytes copied, 0.00060265 s, 850 kB/s 2022-04-02T03:08:20.650 DEBUG:teuthology.orchestra.run.smithi104:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2022-04-02T03:08:20.656 INFO:teuthology.orchestra.run.smithi104.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2022-04-02T03:08:20.657 INFO:teuthology.orchestra.run.smithi104.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-04-02T03:08:20.657 INFO:teuthology.orchestra.run.smithi104.stdout:Device: 5h/5d Inode: 514 Links: 1 2022-04-02T03:08:20.657 INFO:teuthology.orchestra.run.smithi104.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-04-02T03:08:20.657 INFO:teuthology.orchestra.run.smithi104.stdout:Access: 2022-04-02 03:08:17.295551728 +0000 2022-04-02T03:08:20.658 INFO:teuthology.orchestra.run.smithi104.stdout:Modify: 2022-04-02 03:08:17.295551728 +0000 2022-04-02T03:08:20.658 INFO:teuthology.orchestra.run.smithi104.stdout:Change: 2022-04-02 03:08:17.295551728 +0000 2022-04-02T03:08:20.658 INFO:teuthology.orchestra.run.smithi104.stdout: Birth: - 2022-04-02T03:08:20.662 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records in 2022-04-02T03:08:20.662 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records out 2022-04-02T03:08:20.662 INFO:teuthology.orchestra.run.smithi104.stderr:512 bytes copied, 0.000508713 s, 1.0 MB/s 2022-04-02T03:08:20.665 DEBUG:teuthology.orchestra.run.smithi104:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2022-04-02T03:08:20.713 INFO:teuthology.orchestra.run.smithi104.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2022-04-02T03:08:20.713 INFO:teuthology.orchestra.run.smithi104.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-04-02T03:08:20.713 INFO:teuthology.orchestra.run.smithi104.stdout:Device: 5h/5d Inode: 518 Links: 1 2022-04-02T03:08:20.714 INFO:teuthology.orchestra.run.smithi104.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-04-02T03:08:20.714 INFO:teuthology.orchestra.run.smithi104.stdout:Access: 2022-04-02 03:08:17.323551728 +0000 2022-04-02T03:08:20.714 INFO:teuthology.orchestra.run.smithi104.stdout:Modify: 2022-04-02 03:08:17.323551728 +0000 2022-04-02T03:08:20.714 INFO:teuthology.orchestra.run.smithi104.stdout:Change: 2022-04-02 03:08:17.323551728 +0000 2022-04-02T03:08:20.715 INFO:teuthology.orchestra.run.smithi104.stdout: Birth: - 2022-04-02T03:08:20.721 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records in 2022-04-02T03:08:20.722 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records out 2022-04-02T03:08:20.722 INFO:teuthology.orchestra.run.smithi104.stderr:512 bytes copied, 0.000587678 s, 871 kB/s 2022-04-02T03:08:20.725 DEBUG:teuthology.orchestra.run.smithi104:> sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2022-04-02T03:08:21.031 INFO:teuthology.orchestra.run.smithi104.stdout:loop 2022-04-02T03:08:21.032 INFO:tasks.nvme_loop:Connecting nvme_loop smithi104:/dev/vg_nvme/lv_1... 2022-04-02T03:08:21.032 DEBUG:teuthology.orchestra.run.smithi104:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2022-04-02T03:08:21.053 INFO:teuthology.orchestra.run.smithi104.stdout:1 2022-04-02T03:08:21.078 INFO:teuthology.orchestra.run.smithi104.stdout:/dev/vg_nvme/lv_11 2022-04-02T03:08:21.093 INFO:tasks.nvme_loop:Connecting nvme_loop smithi104:/dev/vg_nvme/lv_2... 2022-04-02T03:08:21.093 DEBUG:teuthology.orchestra.run.smithi104:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2022-04-02T03:08:21.152 INFO:teuthology.orchestra.run.smithi104.stdout:1 2022-04-02T03:08:21.169 INFO:teuthology.orchestra.run.smithi104.stdout:/dev/vg_nvme/lv_21 2022-04-02T03:08:21.186 INFO:tasks.nvme_loop:Connecting nvme_loop smithi104:/dev/vg_nvme/lv_3... 2022-04-02T03:08:21.187 DEBUG:teuthology.orchestra.run.smithi104:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2022-04-02T03:08:21.206 INFO:teuthology.orchestra.run.smithi104.stdout:1 2022-04-02T03:08:21.227 INFO:teuthology.orchestra.run.smithi104.stdout:/dev/vg_nvme/lv_31 2022-04-02T03:08:21.246 INFO:tasks.nvme_loop:Connecting nvme_loop smithi104:/dev/vg_nvme/lv_4... 2022-04-02T03:08:21.247 DEBUG:teuthology.orchestra.run.smithi104:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2022-04-02T03:08:21.265 INFO:teuthology.orchestra.run.smithi104.stdout:1 2022-04-02T03:08:21.286 INFO:teuthology.orchestra.run.smithi104.stdout:/dev/vg_nvme/lv_41 2022-04-02T03:08:21.304 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2022-04-02T03:08:21.304 DEBUG:teuthology.orchestra.run.smithi104:> dd if=/scratch_devs of=/dev/stdout 2022-04-02T03:08:21.353 DEBUG:teuthology.orchestra.run.smithi104:> sudo nvme list 2022-04-02T03:08:21.363 INFO:teuthology.orchestra.run.smithi104.stdout:Node SN Model Namespace Usage Format FW Rev 2022-04-02T03:08:21.363 INFO:teuthology.orchestra.run.smithi104.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2022-04-02T03:08:21.363 INFO:teuthology.orchestra.run.smithi104.stdout:/dev/nvme0n1 PHFT620400NW400BGN INTEL SSDPEDMD400G4 1 400.09 GB / 400.09 GB 512 B + 0 B 8DV101H0 2022-04-02T03:08:21.363 INFO:teuthology.orchestra.run.smithi104.stdout:/dev/nvme1n1 00d0548de1b3129c Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-04-02T03:08:21.364 INFO:teuthology.orchestra.run.smithi104.stdout:/dev/nvme2n1 c2b45a86916e8ee8 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-04-02T03:08:21.364 INFO:teuthology.orchestra.run.smithi104.stdout:/dev/nvme3n1 27638110f1cb2b7a Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-04-02T03:08:21.364 INFO:teuthology.orchestra.run.smithi104.stdout:/dev/nvme4n1 dc9f91e5e480c48f Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-04-02T03:08:21.365 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2022-04-02T03:08:21.365 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2022-04-02T03:08:21.366 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/scratch_devs 2022-04-02T03:08:21.381 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2022-04-02T03:08:21.382 DEBUG:teuthology.orchestra.run.smithi142:> dd if=/scratch_devs of=/dev/stdout 2022-04-02T03:08:21.389 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-04-02T03:08:21.390 DEBUG:teuthology.orchestra.run.smithi142:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2022-04-02T03:08:21.397 INFO:teuthology.orchestra.run.smithi142.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2022-04-02T03:08:21.397 INFO:teuthology.orchestra.run.smithi142.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-04-02T03:08:21.397 INFO:teuthology.orchestra.run.smithi142.stdout:Device: 5h/5d Inode: 499 Links: 1 2022-04-02T03:08:21.397 INFO:teuthology.orchestra.run.smithi142.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-04-02T03:08:21.398 INFO:teuthology.orchestra.run.smithi142.stdout:Access: 2022-04-02 03:08:17.410054346 +0000 2022-04-02T03:08:21.398 INFO:teuthology.orchestra.run.smithi142.stdout:Modify: 2022-04-02 03:08:17.410054346 +0000 2022-04-02T03:08:21.398 INFO:teuthology.orchestra.run.smithi142.stdout:Change: 2022-04-02 03:08:17.410054346 +0000 2022-04-02T03:08:21.398 INFO:teuthology.orchestra.run.smithi142.stdout: Birth: - 2022-04-02T03:08:21.408 INFO:teuthology.orchestra.run.smithi142.stderr:1+0 records in 2022-04-02T03:08:21.409 INFO:teuthology.orchestra.run.smithi142.stderr:1+0 records out 2022-04-02T03:08:21.409 INFO:teuthology.orchestra.run.smithi142.stderr:512 bytes copied, 0.00060445 s, 847 kB/s 2022-04-02T03:08:21.411 DEBUG:teuthology.orchestra.run.smithi142:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2022-04-02T03:08:21.417 INFO:teuthology.orchestra.run.smithi142.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2022-04-02T03:08:21.417 INFO:teuthology.orchestra.run.smithi142.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-04-02T03:08:21.417 INFO:teuthology.orchestra.run.smithi142.stdout:Device: 5h/5d Inode: 507 Links: 1 2022-04-02T03:08:21.417 INFO:teuthology.orchestra.run.smithi142.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-04-02T03:08:21.418 INFO:teuthology.orchestra.run.smithi142.stdout:Access: 2022-04-02 03:08:17.438054346 +0000 2022-04-02T03:08:21.418 INFO:teuthology.orchestra.run.smithi142.stdout:Modify: 2022-04-02 03:08:17.438054346 +0000 2022-04-02T03:08:21.418 INFO:teuthology.orchestra.run.smithi142.stdout:Change: 2022-04-02 03:08:17.438054346 +0000 2022-04-02T03:08:21.418 INFO:teuthology.orchestra.run.smithi142.stdout: Birth: - 2022-04-02T03:08:21.424 INFO:teuthology.orchestra.run.smithi142.stderr:1+0 records in 2022-04-02T03:08:21.425 INFO:teuthology.orchestra.run.smithi142.stderr:1+0 records out 2022-04-02T03:08:21.425 INFO:teuthology.orchestra.run.smithi142.stderr:512 bytes copied, 0.00046276 s, 1.1 MB/s 2022-04-02T03:08:21.428 DEBUG:teuthology.orchestra.run.smithi142:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2022-04-02T03:08:21.480 INFO:teuthology.orchestra.run.smithi142.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2022-04-02T03:08:21.480 INFO:teuthology.orchestra.run.smithi142.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-04-02T03:08:21.480 INFO:teuthology.orchestra.run.smithi142.stdout:Device: 5h/5d Inode: 514 Links: 1 2022-04-02T03:08:21.481 INFO:teuthology.orchestra.run.smithi142.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-04-02T03:08:21.481 INFO:teuthology.orchestra.run.smithi142.stdout:Access: 2022-04-02 03:08:17.470054345 +0000 2022-04-02T03:08:21.481 INFO:teuthology.orchestra.run.smithi142.stdout:Modify: 2022-04-02 03:08:17.466054345 +0000 2022-04-02T03:08:21.482 INFO:teuthology.orchestra.run.smithi142.stdout:Change: 2022-04-02 03:08:17.466054345 +0000 2022-04-02T03:08:21.482 INFO:teuthology.orchestra.run.smithi142.stdout: Birth: - 2022-04-02T03:08:21.487 INFO:teuthology.orchestra.run.smithi142.stderr:1+0 records in 2022-04-02T03:08:21.488 INFO:teuthology.orchestra.run.smithi142.stderr:1+0 records out 2022-04-02T03:08:21.488 INFO:teuthology.orchestra.run.smithi142.stderr:512 bytes copied, 0.000493268 s, 1.0 MB/s 2022-04-02T03:08:21.491 DEBUG:teuthology.orchestra.run.smithi142:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2022-04-02T03:08:21.497 INFO:teuthology.orchestra.run.smithi142.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2022-04-02T03:08:21.497 INFO:teuthology.orchestra.run.smithi142.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-04-02T03:08:21.498 INFO:teuthology.orchestra.run.smithi142.stdout:Device: 5h/5d Inode: 519 Links: 1 2022-04-02T03:08:21.498 INFO:teuthology.orchestra.run.smithi142.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-04-02T03:08:21.498 INFO:teuthology.orchestra.run.smithi142.stdout:Access: 2022-04-02 03:08:17.494054345 +0000 2022-04-02T03:08:21.498 INFO:teuthology.orchestra.run.smithi142.stdout:Modify: 2022-04-02 03:08:17.494054345 +0000 2022-04-02T03:08:21.499 INFO:teuthology.orchestra.run.smithi142.stdout:Change: 2022-04-02 03:08:17.494054345 +0000 2022-04-02T03:08:21.499 INFO:teuthology.orchestra.run.smithi142.stdout: Birth: - 2022-04-02T03:08:21.504 INFO:teuthology.orchestra.run.smithi142.stderr:1+0 records in 2022-04-02T03:08:21.504 INFO:teuthology.orchestra.run.smithi142.stderr:1+0 records out 2022-04-02T03:08:21.504 INFO:teuthology.orchestra.run.smithi142.stderr:512 bytes copied, 0.000575584 s, 890 kB/s 2022-04-02T03:08:21.508 DEBUG:teuthology.orchestra.run.smithi142:> sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2022-04-02T03:08:21.593 INFO:teuthology.orchestra.run.smithi142.stdout:loop 2022-04-02T03:08:21.595 INFO:tasks.nvme_loop:Connecting nvme_loop smithi142:/dev/vg_nvme/lv_1... 2022-04-02T03:08:21.595 DEBUG:teuthology.orchestra.run.smithi142:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2022-04-02T03:08:21.616 INFO:teuthology.orchestra.run.smithi142.stdout:1 2022-04-02T03:08:21.638 INFO:teuthology.orchestra.run.smithi142.stdout:/dev/vg_nvme/lv_11 2022-04-02T03:08:21.661 INFO:tasks.nvme_loop:Connecting nvme_loop smithi142:/dev/vg_nvme/lv_2... 2022-04-02T03:08:21.662 DEBUG:teuthology.orchestra.run.smithi142:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2022-04-02T03:08:21.683 INFO:teuthology.orchestra.run.smithi142.stdout:1 2022-04-02T03:08:21.706 INFO:teuthology.orchestra.run.smithi142.stdout:/dev/vg_nvme/lv_21 2022-04-02T03:08:21.731 INFO:tasks.nvme_loop:Connecting nvme_loop smithi142:/dev/vg_nvme/lv_3... 2022-04-02T03:08:21.731 DEBUG:teuthology.orchestra.run.smithi142:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2022-04-02T03:08:21.752 INFO:teuthology.orchestra.run.smithi142.stdout:1 2022-04-02T03:08:21.774 INFO:teuthology.orchestra.run.smithi142.stdout:/dev/vg_nvme/lv_31 2022-04-02T03:08:21.798 INFO:tasks.nvme_loop:Connecting nvme_loop smithi142:/dev/vg_nvme/lv_4... 2022-04-02T03:08:21.799 DEBUG:teuthology.orchestra.run.smithi142:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2022-04-02T03:08:21.864 INFO:teuthology.orchestra.run.smithi142.stdout:1 2022-04-02T03:08:21.890 INFO:teuthology.orchestra.run.smithi142.stdout:/dev/vg_nvme/lv_41 2022-04-02T03:08:21.916 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2022-04-02T03:08:21.917 DEBUG:teuthology.orchestra.run.smithi142:> dd if=/scratch_devs of=/dev/stdout 2022-04-02T03:08:21.926 DEBUG:teuthology.orchestra.run.smithi142:> sudo nvme list 2022-04-02T03:08:21.981 INFO:teuthology.orchestra.run.smithi142.stdout:Node SN Model Namespace Usage Format FW Rev 2022-04-02T03:08:21.981 INFO:teuthology.orchestra.run.smithi142.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2022-04-02T03:08:21.981 INFO:teuthology.orchestra.run.smithi142.stdout:/dev/nvme0n1 CVFT6303009Y400BGN INTEL SSDPEDMD400G4 1 400.09 GB / 400.09 GB 512 B + 0 B 8DV101H0 2022-04-02T03:08:21.982 INFO:teuthology.orchestra.run.smithi142.stdout:/dev/nvme1n1 9c349759eb5f3909 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-04-02T03:08:21.982 INFO:teuthology.orchestra.run.smithi142.stdout:/dev/nvme2n1 32e0eeabce704b9b Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-04-02T03:08:21.982 INFO:teuthology.orchestra.run.smithi142.stdout:/dev/nvme3n1 40656fcbb9e219e4 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-04-02T03:08:21.982 INFO:teuthology.orchestra.run.smithi142.stdout:/dev/nvme4n1 e5fc22fd9ba408ad Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-04-02T03:08:21.983 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2022-04-02T03:08:21.984 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2022-04-02T03:08:21.984 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd of=/scratch_devs 2022-04-02T03:08:21.999 INFO:teuthology.run_tasks:Running task rook... 2022-04-02T03:08:22.075 INFO:tasks.rook:Rook start 2022-04-02T03:08:22.076 INFO:tasks.rook:Config: {'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a', 'rook_image': 'rook/ceph:master'} 2022-04-02T03:08:22.076 INFO:tasks.rook:Ceph image is quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a 2022-04-02T03:08:22.076 INFO:tasks.rook:Cloning https://github.com/rook/rook branch master 2022-04-02T03:08:22.077 DEBUG:teuthology.orchestra.run.smithi029:> rm -rf rook && git clone --single-branch --branch master https://github.com/rook/rook rook 2022-04-02T03:08:22.086 INFO:teuthology.orchestra.run.smithi029.stderr:Cloning into 'rook'... 2022-04-02T03:08:25.136 INFO:tasks.rook:/home/teuthworker 2022-04-02T03:08:25.137 DEBUG:teuthology.orchestra.run.smithi029:> PATH=/sbin:/usr/sbin ip addr show 2022-04-02T03:08:25.145 INFO:teuthology.orchestra.run.smithi029.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2022-04-02T03:08:25.145 INFO:teuthology.orchestra.run.smithi029.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2022-04-02T03:08:25.145 INFO:teuthology.orchestra.run.smithi029.stdout: inet 127.0.0.1/8 scope host lo 2022-04-02T03:08:25.146 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2022-04-02T03:08:25.146 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 ::1/128 scope host 2022-04-02T03:08:25.146 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2022-04-02T03:08:25.146 INFO:teuthology.orchestra.run.smithi029.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2022-04-02T03:08:25.147 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:6b:fd:a0 brd ff:ff:ff:ff:ff:ff 2022-04-02T03:08:25.147 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp7s0f0 2022-04-02T03:08:25.147 INFO:teuthology.orchestra.run.smithi029.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2022-04-02T03:08:25.147 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:6b:fd:a1 brd ff:ff:ff:ff:ff:ff 2022-04-02T03:08:25.147 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp7s0f1 2022-04-02T03:08:25.148 INFO:teuthology.orchestra.run.smithi029.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2022-04-02T03:08:25.148 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:bd:12:5a brd ff:ff:ff:ff:ff:ff 2022-04-02T03:08:25.148 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp1s0f0 2022-04-02T03:08:25.148 INFO:teuthology.orchestra.run.smithi029.stdout: inet 172.21.15.29/20 brd 172.21.15.255 scope global dynamic ens1f0 2022-04-02T03:08:25.149 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft 42839sec preferred_lft 42839sec 2022-04-02T03:08:25.149 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::ec4:7aff:febd:125a/64 scope link 2022-04-02T03:08:25.149 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2022-04-02T03:08:25.149 INFO:teuthology.orchestra.run.smithi029.stdout:5: ens1f1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2022-04-02T03:08:25.150 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:bd:12:5b brd ff:ff:ff:ff:ff:ff 2022-04-02T03:08:25.150 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp1s0f1 2022-04-02T03:08:25.150 INFO:teuthology.orchestra.run.smithi029.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2022-04-02T03:08:25.150 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 02:42:77:fa:de:a2 brd ff:ff:ff:ff:ff:ff 2022-04-02T03:08:25.150 INFO:teuthology.orchestra.run.smithi029.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2022-04-02T03:08:25.151 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2022-04-02T03:08:25.152 DEBUG:teuthology.orchestra.run.smithi029:> which systemctl 2022-04-02T03:08:25.157 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/systemctl 2022-04-02T03:08:25.157 DEBUG:teuthology.orchestra.run.smithi029:> true 2022-04-02T03:08:25.230 DEBUG:teuthology.orchestra.run.smithi029:> sudo lsb_release -is 2022-04-02T03:08:25.267 INFO:teuthology.orchestra.run.smithi029.stdout:Ubuntu 2022-04-02T03:08:25.273 DEBUG:teuthology.misc:System to be installed: Ubuntu 2022-04-02T03:08:25.273 INFO:tasks.rook:['__class__', '__del__', '__delattr__', '__dir__', '__eq__', '__format__', '__ge__', '__getattribute__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__le__', '__lt__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '_format_size', '_reconnect', '_runner', '_set_iface_and_cidr', '_sftp_get_file', '_sftp_get_size', '_sftp_open_file', '_sftp_put_file', 'chcon', 'chmod', 'connect', 'copy_file', 'ensure_online', 'get_file', 'get_tar', 'get_tar_stream', 'mkdtemp', 'mktemp', 'move_file', 'put_file', 'read_file', 'reconnect', 'remove', 'run', 'sh', 'sh_file', 'sudo_write_file', 'write_file'] 2022-04-02T03:08:25.327 DEBUG:teuthology.orchestra.remote:smithi029:rook/deploy/examples/operator.yaml is 20KB 2022-04-02T03:08:25.342 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2022-04-02T03:08:25.342 DEBUG:teuthology.orchestra.run.smithi029:> dd if=rook/deploy/examples/operator.yaml of=/dev/stdout 2022-04-02T03:08:25.351 INFO:tasks.rook:Patching operator to use image rook/ceph:master 2022-04-02T03:08:25.393 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2022-04-02T03:08:25.393 DEBUG:teuthology.orchestra.run.smithi029:> dd of=operator.yaml 2022-04-02T03:08:25.407 INFO:tasks.rook:Deploying operator 2022-04-02T03:08:25.416 DEBUG:teuthology.orchestra.remote:smithi029:rook/deploy/examples/operator.yaml is 20KB 2022-04-02T03:08:25.434 DEBUG:teuthology.orchestra.remote:smithi029:rook/deploy/examples/operator.yaml is 20KB 2022-04-02T03:08:25.444 DEBUG:teuthology.orchestra.run.smithi029:> kubectl create -f rook/deploy/examples/crds.yaml -f rook/deploy/examples/common.yaml -f operator.yaml 2022-04-02T03:08:25.789 INFO:teuthology.orchestra.run.smithi029.stdout:customresourcedefinition.apiextensions.k8s.io/cephblockpools.ceph.rook.io created 2022-04-02T03:08:25.948 INFO:teuthology.orchestra.run.smithi029.stdout:customresourcedefinition.apiextensions.k8s.io/cephbucketnotifications.ceph.rook.io created 2022-04-02T03:08:25.972 INFO:teuthology.orchestra.run.smithi029.stdout:customresourcedefinition.apiextensions.k8s.io/cephbuckettopics.ceph.rook.io created 2022-04-02T03:08:25.999 INFO:teuthology.orchestra.run.smithi029.stdout:customresourcedefinition.apiextensions.k8s.io/cephclients.ceph.rook.io created 2022-04-02T03:08:26.144 INFO:teuthology.orchestra.run.smithi029.stdout:customresourcedefinition.apiextensions.k8s.io/cephclusters.ceph.rook.io created 2022-04-02T03:08:26.181 INFO:teuthology.orchestra.run.smithi029.stdout:customresourcedefinition.apiextensions.k8s.io/cephfilesystemmirrors.ceph.rook.io created 2022-04-02T03:08:26.294 INFO:teuthology.orchestra.run.smithi029.stdout:customresourcedefinition.apiextensions.k8s.io/cephfilesystems.ceph.rook.io created 2022-04-02T03:08:26.329 INFO:teuthology.orchestra.run.smithi029.stdout:customresourcedefinition.apiextensions.k8s.io/cephfilesystemsubvolumegroups.ceph.rook.io created 2022-04-02T03:08:26.383 INFO:teuthology.orchestra.run.smithi029.stdout:customresourcedefinition.apiextensions.k8s.io/cephnfses.ceph.rook.io created 2022-04-02T03:08:26.497 INFO:teuthology.orchestra.run.smithi029.stdout:customresourcedefinition.apiextensions.k8s.io/cephobjectrealms.ceph.rook.io created 2022-04-02T03:08:26.563 INFO:teuthology.orchestra.run.smithi029.stdout:customresourcedefinition.apiextensions.k8s.io/cephobjectstores.ceph.rook.io created 2022-04-02T03:08:26.708 INFO:teuthology.orchestra.run.smithi029.stdout:customresourcedefinition.apiextensions.k8s.io/cephobjectstoreusers.ceph.rook.io created 2022-04-02T03:08:26.716 INFO:teuthology.orchestra.run.smithi029.stdout:customresourcedefinition.apiextensions.k8s.io/cephobjectzonegroups.ceph.rook.io created 2022-04-02T03:08:26.742 INFO:teuthology.orchestra.run.smithi029.stdout:customresourcedefinition.apiextensions.k8s.io/cephobjectzones.ceph.rook.io created 2022-04-02T03:08:26.805 INFO:teuthology.orchestra.run.smithi029.stdout:customresourcedefinition.apiextensions.k8s.io/cephrbdmirrors.ceph.rook.io created 2022-04-02T03:08:26.916 INFO:teuthology.orchestra.run.smithi029.stdout:customresourcedefinition.apiextensions.k8s.io/objectbucketclaims.objectbucket.io created 2022-04-02T03:08:26.945 INFO:teuthology.orchestra.run.smithi029.stdout:customresourcedefinition.apiextensions.k8s.io/objectbuckets.objectbucket.io created 2022-04-02T03:08:27.071 INFO:teuthology.orchestra.run.smithi029.stdout:namespace/rook-ceph created 2022-04-02T03:08:27.122 INFO:teuthology.orchestra.run.smithi029.stdout:clusterrole.rbac.authorization.k8s.io/cephfs-csi-nodeplugin created 2022-04-02T03:08:27.232 INFO:teuthology.orchestra.run.smithi029.stdout:clusterrole.rbac.authorization.k8s.io/cephfs-external-provisioner-runner created 2022-04-02T03:08:27.246 INFO:teuthology.orchestra.run.smithi029.stdout:clusterrole.rbac.authorization.k8s.io/psp:rook created 2022-04-02T03:08:27.272 INFO:teuthology.orchestra.run.smithi029.stdout:clusterrole.rbac.authorization.k8s.io/rbd-csi-nodeplugin created 2022-04-02T03:08:27.284 INFO:teuthology.orchestra.run.smithi029.stdout:clusterrole.rbac.authorization.k8s.io/rbd-external-provisioner-runner created 2022-04-02T03:08:27.318 INFO:teuthology.orchestra.run.smithi029.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-cluster-mgmt created 2022-04-02T03:08:27.403 INFO:teuthology.orchestra.run.smithi029.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-global created 2022-04-02T03:08:27.435 INFO:teuthology.orchestra.run.smithi029.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-mgr-cluster created 2022-04-02T03:08:27.453 INFO:teuthology.orchestra.run.smithi029.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-mgr-system created 2022-04-02T03:08:27.502 INFO:teuthology.orchestra.run.smithi029.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-object-bucket created 2022-04-02T03:08:27.571 INFO:teuthology.orchestra.run.smithi029.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-osd created 2022-04-02T03:08:27.593 INFO:teuthology.orchestra.run.smithi029.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-system created 2022-04-02T03:08:27.622 INFO:teuthology.orchestra.run.smithi029.stdout:clusterrolebinding.rbac.authorization.k8s.io/cephfs-csi-nodeplugin created 2022-04-02T03:08:27.634 INFO:teuthology.orchestra.run.smithi029.stdout:clusterrolebinding.rbac.authorization.k8s.io/cephfs-csi-provisioner-role created 2022-04-02T03:08:27.709 INFO:teuthology.orchestra.run.smithi029.stdout:clusterrolebinding.rbac.authorization.k8s.io/rbd-csi-nodeplugin created 2022-04-02T03:08:27.712 INFO:teuthology.orchestra.run.smithi029.stdout:clusterrolebinding.rbac.authorization.k8s.io/rbd-csi-provisioner-role created 2022-04-02T03:08:27.718 INFO:teuthology.orchestra.run.smithi029.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-global created 2022-04-02T03:08:27.725 INFO:teuthology.orchestra.run.smithi029.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-mgr-cluster created 2022-04-02T03:08:27.742 INFO:teuthology.orchestra.run.smithi029.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-object-bucket created 2022-04-02T03:08:27.800 INFO:teuthology.orchestra.run.smithi029.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-osd created 2022-04-02T03:08:27.862 INFO:teuthology.orchestra.run.smithi029.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-system created 2022-04-02T03:08:27.866 INFO:teuthology.orchestra.run.smithi029.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-system-psp created 2022-04-02T03:08:27.893 INFO:teuthology.orchestra.run.smithi029.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-csi-cephfs-plugin-sa-psp created 2022-04-02T03:08:27.925 INFO:teuthology.orchestra.run.smithi029.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-csi-cephfs-provisioner-sa-psp created 2022-04-02T03:08:27.944 INFO:teuthology.orchestra.run.smithi029.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-csi-rbd-plugin-sa-psp created 2022-04-02T03:08:28.001 INFO:teuthology.orchestra.run.smithi029.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-csi-rbd-provisioner-sa-psp created 2022-04-02T03:08:28.007 INFO:teuthology.orchestra.run.smithi029.stderr:Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+ 2022-04-02T03:08:28.007 INFO:teuthology.orchestra.run.smithi029.stdout:podsecuritypolicy.policy/00-rook-privileged created 2022-04-02T03:08:28.017 INFO:teuthology.orchestra.run.smithi029.stdout:role.rbac.authorization.k8s.io/cephfs-external-provisioner-cfg created 2022-04-02T03:08:28.035 INFO:teuthology.orchestra.run.smithi029.stdout:role.rbac.authorization.k8s.io/rbd-csi-nodeplugin created 2022-04-02T03:08:28.087 INFO:teuthology.orchestra.run.smithi029.stdout:role.rbac.authorization.k8s.io/rbd-external-provisioner-cfg created 2022-04-02T03:08:28.146 INFO:teuthology.orchestra.run.smithi029.stdout:role.rbac.authorization.k8s.io/rook-ceph-cmd-reporter created 2022-04-02T03:08:28.150 INFO:teuthology.orchestra.run.smithi029.stdout:role.rbac.authorization.k8s.io/rook-ceph-mgr created 2022-04-02T03:08:28.159 INFO:teuthology.orchestra.run.smithi029.stdout:role.rbac.authorization.k8s.io/rook-ceph-osd created 2022-04-02T03:08:28.184 INFO:teuthology.orchestra.run.smithi029.stdout:role.rbac.authorization.k8s.io/rook-ceph-purge-osd created 2022-04-02T03:08:28.233 INFO:teuthology.orchestra.run.smithi029.stdout:role.rbac.authorization.k8s.io/rook-ceph-system created 2022-04-02T03:08:28.279 INFO:teuthology.orchestra.run.smithi029.stdout:rolebinding.rbac.authorization.k8s.io/cephfs-csi-provisioner-role-cfg created 2022-04-02T03:08:28.282 INFO:teuthology.orchestra.run.smithi029.stdout:rolebinding.rbac.authorization.k8s.io/rbd-csi-nodeplugin-role-cfg created 2022-04-02T03:08:28.310 INFO:teuthology.orchestra.run.smithi029.stdout:rolebinding.rbac.authorization.k8s.io/rbd-csi-provisioner-role-cfg created 2022-04-02T03:08:28.318 INFO:teuthology.orchestra.run.smithi029.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-cluster-mgmt created 2022-04-02T03:08:28.335 INFO:teuthology.orchestra.run.smithi029.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-cmd-reporter created 2022-04-02T03:08:28.488 INFO:teuthology.orchestra.run.smithi029.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-cmd-reporter-psp created 2022-04-02T03:08:28.535 INFO:teuthology.orchestra.run.smithi029.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-default-psp created 2022-04-02T03:08:28.879 INFO:teuthology.orchestra.run.smithi029.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-mgr created 2022-04-02T03:08:28.960 INFO:teuthology.orchestra.run.smithi029.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-mgr-psp created 2022-04-02T03:08:29.338 INFO:teuthology.orchestra.run.smithi029.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-mgr-system created 2022-04-02T03:08:29.340 INFO:teuthology.orchestra.run.smithi029.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-osd created 2022-04-02T03:08:29.368 INFO:teuthology.orchestra.run.smithi029.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-osd-psp created 2022-04-02T03:08:30.025 INFO:teuthology.orchestra.run.smithi029.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-purge-osd created 2022-04-02T03:08:30.099 INFO:teuthology.orchestra.run.smithi029.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-system created 2022-04-02T03:08:30.581 INFO:teuthology.orchestra.run.smithi029.stdout:serviceaccount/rook-ceph-cmd-reporter created 2022-04-02T03:08:30.588 INFO:teuthology.orchestra.run.smithi029.stdout:serviceaccount/rook-ceph-mgr created 2022-04-02T03:08:30.611 INFO:teuthology.orchestra.run.smithi029.stdout:serviceaccount/rook-ceph-osd created 2022-04-02T03:08:30.642 INFO:teuthology.orchestra.run.smithi029.stdout:serviceaccount/rook-ceph-purge-osd created 2022-04-02T03:08:30.836 INFO:teuthology.orchestra.run.smithi029.stdout:serviceaccount/rook-ceph-system created 2022-04-02T03:08:30.932 INFO:teuthology.orchestra.run.smithi029.stdout:serviceaccount/rook-csi-cephfs-plugin-sa created 2022-04-02T03:08:31.013 INFO:teuthology.orchestra.run.smithi029.stdout:serviceaccount/rook-csi-cephfs-provisioner-sa created 2022-04-02T03:08:31.023 INFO:teuthology.orchestra.run.smithi029.stdout:serviceaccount/rook-csi-rbd-plugin-sa created 2022-04-02T03:08:31.059 INFO:teuthology.orchestra.run.smithi029.stdout:serviceaccount/rook-csi-rbd-provisioner-sa created 2022-04-02T03:08:31.100 INFO:teuthology.orchestra.run.smithi029.stdout:configmap/rook-ceph-operator-config created 2022-04-02T03:08:31.179 INFO:teuthology.orchestra.run.smithi029.stdout:deployment.apps/rook-ceph-operator created 2022-04-02T03:08:31.186 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-04-02T03:08:31.282 INFO:teuthology.orchestra.run.smithi029.stdout:NAME READY STATUS RESTARTS AGE 2022-04-02T03:08:31.282 INFO:teuthology.orchestra.run.smithi029.stdout:rook-ceph-operator-75dd789779-p9bzh 0/1 Pending 0 0s 2022-04-02T03:08:41.284 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-04-02T03:08:41.351 INFO:teuthology.orchestra.run.smithi029.stdout:NAME READY STATUS RESTARTS AGE 2022-04-02T03:08:41.351 INFO:teuthology.orchestra.run.smithi029.stdout:rook-ceph-operator-75dd789779-p9bzh 0/1 ContainerCreating 0 10s 2022-04-02T03:08:51.354 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-04-02T03:08:51.426 INFO:teuthology.orchestra.run.smithi029.stdout:NAME READY STATUS RESTARTS AGE 2022-04-02T03:08:51.426 INFO:teuthology.orchestra.run.smithi029.stdout:rook-ceph-operator-75dd789779-p9bzh 0/1 ContainerCreating 0 20s 2022-04-02T03:09:01.429 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-04-02T03:09:01.494 INFO:teuthology.orchestra.run.smithi029.stdout:NAME READY STATUS RESTARTS AGE 2022-04-02T03:09:01.494 INFO:teuthology.orchestra.run.smithi029.stdout:rook-ceph-operator-75dd789779-p9bzh 0/1 ContainerCreating 0 30s 2022-04-02T03:09:11.498 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-04-02T03:09:11.562 INFO:teuthology.orchestra.run.smithi029.stdout:NAME READY STATUS RESTARTS AGE 2022-04-02T03:09:11.563 INFO:teuthology.orchestra.run.smithi029.stdout:rook-ceph-operator-75dd789779-p9bzh 1/1 Running 0 40s 2022-04-02T03:09:11.564 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph logs -f rook-ceph-operator-75dd789779-p9bzh 2022-04-02T03:09:11.573 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2022-04-02T03:09:11.574 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/scratch_devs of=/dev/stdout 2022-04-02T03:09:11.581 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2022-04-02T03:09:11.581 DEBUG:teuthology.orchestra.run.smithi104:> dd if=/scratch_devs of=/dev/stdout 2022-04-02T03:09:11.589 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2022-04-02T03:09:11.590 DEBUG:teuthology.orchestra.run.smithi142:> dd if=/scratch_devs of=/dev/stdout 2022-04-02T03:09:11.600 INFO:tasks.rook: override: [mgr] debug mgr = 20 2022-04-02T03:09:11.600 INFO:tasks.rook: override: [mgr] debug ms = 1 2022-04-02T03:09:11.600 INFO:tasks.rook: override: [mon] debug mon = 20 2022-04-02T03:09:11.601 INFO:tasks.rook: override: [mon] debug ms = 1 2022-04-02T03:09:11.601 INFO:tasks.rook: override: [mon] debug paxos = 20 2022-04-02T03:09:11.601 INFO:tasks.rook: override: [osd] debug ms = 1 2022-04-02T03:09:11.602 INFO:tasks.rook: override: [osd] debug osd = 20 2022-04-02T03:09:11.602 INFO:tasks.rook:Config: b"[global]\n\nlog to file = true\n\nmon clock drift allowed = 1.000\n\n# replicate across OSDs, not hosts\nosd crush chooseleaf type = 0\n\n# enable some debugging\nauth debug = true\nms die on old message = true\nms die on bug = true\ndebug asserts on shutdown = true\n\n\n[osd]\n# debugging\nosd debug shutdown = true\nosd debug op order = true\nosd debug verify stray on activate = true\nosd debug pg log writeout = true\nosd debug verify cached snaps = true\nosd debug verify missing on start = true\nosd debug misdirected ops = true\nosd op queue = debug_random\nosd op queue cut off = debug_random\nosd shutdown pgref assert = true\nbdev debug aio = true\nosd sloppy crc = true\ndebug ms = 1\ndebug osd = 20\n\n\n[mon]\n# rotate auth tickets quickly to exercise renewal paths\nauth mon ticket ttl = 660# 11m\nauth service ticket ttl = 240# 4m\n\n# don't complain about global id reclaim\nmon_warn_on_insecure_global_id_reclaim = false\nmon_warn_on_insecure_global_id_reclaim_allowed = false\ndebug mon = 20\ndebug ms = 1\ndebug paxos = 20\n[mgr]\ndebug mgr = 20\ndebug ms = 1\n" 2022-04-02T03:09:11.604 DEBUG:teuthology.orchestra.run.smithi029:> kubectl create -f - 2022-04-02T03:09:11.650 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:09:02.232840 I | rookcmd: starting Rook v1.9.0-alpha.0.4.gd847fc86b with arguments '/usr/local/bin/rook ceph operator' 2022-04-02T03:09:11.650 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:09:02.232954 I | rookcmd: flag values: --enable-machine-disruption-budget=false, --help=false, --kubeconfig=, --log-level=INFO, --operator-image=, --service-account= 2022-04-02T03:09:11.650 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:09:02.232958 I | cephcmd: starting Rook-Ceph operator 2022-04-02T03:09:11.650 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:09:02.429738 I | cephcmd: base ceph version inside the rook operator image is "ceph version 16.2.7 (dd0603118f56ab514f133c8d2e3adfc983942503) pacific (stable)" 2022-04-02T03:09:11.651 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:09:02.439126 I | op-k8sutil: ROOK_CURRENT_NAMESPACE_ONLY="false" (env var) 2022-04-02T03:09:11.651 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:09:02.439138 I | operator: watching all namespaces for Ceph CRs 2022-04-02T03:09:11.651 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:09:02.439276 I | operator: setting up schemes 2022-04-02T03:09:11.651 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:09:02.443360 I | operator: setting up the controller-runtime manager 2022-04-02T03:09:11.651 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:09:02.650514 I | operator: looking for admission webhook secret "rook-ceph-admission-controller" 2022-04-02T03:09:11.652 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:09:02.654079 I | operator: admission webhook secret "rook-ceph-admission-controller" not found. proceeding without the admission controller 2022-04-02T03:09:11.652 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:09:02.654124 I | ceph-cluster-controller: successfully started 2022-04-02T03:09:11.652 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:09:02.654192 I | ceph-cluster-controller: enabling hotplug orchestration 2022-04-02T03:09:11.652 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:09:02.654214 I | ceph-crashcollector-controller: successfully started 2022-04-02T03:09:11.653 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:09:02.654243 I | ceph-block-pool-controller: successfully started 2022-04-02T03:09:11.653 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:09:02.654289 I | ceph-object-store-user-controller: successfully started 2022-04-02T03:09:11.653 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:09:02.654314 I | ceph-object-realm-controller: successfully started 2022-04-02T03:09:11.653 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:09:02.654331 I | ceph-object-zonegroup-controller: successfully started 2022-04-02T03:09:11.653 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:09:02.654350 I | ceph-object-zone-controller: successfully started 2022-04-02T03:09:11.655 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:09:02.654504 I | ceph-object-controller: successfully started 2022-04-02T03:09:11.655 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:09:02.654541 I | ceph-file-controller: successfully started 2022-04-02T03:09:11.655 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:09:02.654581 I | ceph-nfs-controller: successfully started 2022-04-02T03:09:11.656 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:09:02.654626 I | ceph-rbd-mirror-controller: successfully started 2022-04-02T03:09:11.656 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:09:02.654670 I | ceph-client-controller: successfully started 2022-04-02T03:09:11.656 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:09:02.654708 I | ceph-filesystem-mirror-controller: successfully started 2022-04-02T03:09:11.656 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:09:02.654752 I | operator: rook-ceph-operator-config-controller successfully started 2022-04-02T03:09:11.656 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:09:02.654774 I | ceph-csi: rook-ceph-operator-csi-controller successfully started 2022-04-02T03:09:11.657 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:09:02.654808 I | op-bucket-prov: rook-ceph-operator-bucket-controller successfully started 2022-04-02T03:09:11.657 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:09:02.654841 I | ceph-bucket-topic: successfully started 2022-04-02T03:09:11.657 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:09:02.654870 I | ceph-bucket-notification: successfully started 2022-04-02T03:09:11.657 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:09:02.654890 I | ceph-bucket-notification: successfully started 2022-04-02T03:09:11.657 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:09:02.654912 I | ceph-fs-subvolumegroup-controller: successfully started 2022-04-02T03:09:11.658 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:09:02.655892 I | operator: starting the controller-runtime manager 2022-04-02T03:09:11.658 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:09:02.758623 I | op-k8sutil: ROOK_CEPH_COMMANDS_TIMEOUT_SECONDS="15" (configmap) 2022-04-02T03:09:11.658 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:09:02.758669 I | op-k8sutil: ROOK_LOG_LEVEL="INFO" (configmap) 2022-04-02T03:09:11.658 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:09:02.758685 I | op-k8sutil: ROOK_ENABLE_DISCOVERY_DAEMON="false" (configmap) 2022-04-02T03:09:11.658 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:09:02.762951 I | operator: rook-ceph-operator-config-controller done reconciling 2022-04-02T03:09:11.659 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:09:02.766714 I | ceph-csi: CSI Ceph RBD driver disabled 2022-04-02T03:09:11.659 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:09:02.766749 I | op-k8sutil: removing daemonset csi-rbdplugin if it exists 2022-04-02T03:09:11.659 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:09:02.769394 I | op-k8sutil: removing deployment csi-rbdplugin-provisioner if it exists 2022-04-02T03:09:11.659 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:09:02.776763 I | ceph-csi: successfully removed CSI Ceph RBD driver 2022-04-02T03:09:11.660 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:09:02.776783 I | ceph-csi: CSI CephFS driver disabled 2022-04-02T03:09:11.660 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:09:02.776791 I | op-k8sutil: removing daemonset csi-cephfsplugin if it exists 2022-04-02T03:09:11.660 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:09:02.779012 I | op-k8sutil: removing deployment csi-cephfsplugin-provisioner if it exists 2022-04-02T03:09:11.660 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:09:02.785717 I | ceph-csi: successfully removed CSI CephFS driver 2022-04-02T03:09:12.544 INFO:teuthology.orchestra.run.smithi029.stdout:configmap/rook-config-override created 2022-04-02T03:09:12.554 INFO:tasks.rook:Cluster: apiVersion: ceph.rook.io/v1 kind: CephCluster metadata: name: rook-ceph namespace: rook-ceph spec: cephVersion: allowUnsupported: true image: quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a dataDirHostPath: /var/lib/rook mgr: count: 1 modules: - enabled: true name: rook mon: allowMultiplePerNode: true count: 3 skipUpgradeChecks: true 2022-04-02T03:09:12.554 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2022-04-02T03:09:12.555 DEBUG:teuthology.orchestra.run.smithi029:> dd of=cluster.yaml 2022-04-02T03:09:12.560 DEBUG:teuthology.orchestra.run.smithi029:> kubectl create -f cluster.yaml 2022-04-02T03:09:14.973 INFO:teuthology.orchestra.run.smithi029.stdout:cephcluster.ceph.rook.io/rook-ceph created 2022-04-02T03:09:14.979 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:09:14.981749 I | ceph-spec: adding finalizer "cephcluster.ceph.rook.io" on "rook-ceph" 2022-04-02T03:09:14.985 DEBUG:teuthology.orchestra.remote:smithi029:rook/deploy/examples/operator.yaml is 20KB 2022-04-02T03:09:14.999 DEBUG:teuthology.orchestra.run.smithi029:> kubectl create -f rook/deploy/examples/toolbox.yaml 2022-04-02T03:09:15.003 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:09:15.005425 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-04-02T03:09:15.026 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:09:15.028835 I | ceph-cluster-controller: clusterInfo not yet found, must be a new cluster. 2022-04-02T03:09:15.042 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:09:15.044776 I | ceph-csi: successfully created csi config map "rook-ceph-csi-config" 2022-04-02T03:09:15.051 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:09:15.053620 I | ceph-spec: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a... 2022-04-02T03:09:15.179 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:09:15.180406 I | op-k8sutil: ROOK_CSI_ENABLE_RBD="true" (configmap) 2022-04-02T03:09:15.180 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:09:15.180444 I | op-k8sutil: ROOK_CSI_ENABLE_CEPHFS="true" (configmap) 2022-04-02T03:09:15.180 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:09:15.180458 I | op-k8sutil: ROOK_CSI_ALLOW_UNSUPPORTED_VERSION="false" (configmap) 2022-04-02T03:09:15.180 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:09:15.180467 I | op-k8sutil: ROOK_CSI_ENABLE_GRPC_METRICS="false" (configmap) 2022-04-02T03:09:15.180 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:09:15.180475 I | op-k8sutil: CSI_ENABLE_HOST_NETWORK="true" (default) 2022-04-02T03:09:15.181 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:09:15.180483 I | op-k8sutil: ROOK_CSI_CEPH_IMAGE="quay.io/cephcsi/cephcsi:v3.5.1" (default) 2022-04-02T03:09:15.181 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:09:15.180492 I | op-k8sutil: ROOK_CSI_REGISTRAR_IMAGE="k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.5.0" (default) 2022-04-02T03:09:15.181 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:09:15.180502 I | op-k8sutil: ROOK_CSI_PROVISIONER_IMAGE="k8s.gcr.io/sig-storage/csi-provisioner:v3.1.0" (default) 2022-04-02T03:09:15.181 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:09:15.180510 I | op-k8sutil: ROOK_CSI_ATTACHER_IMAGE="k8s.gcr.io/sig-storage/csi-attacher:v3.4.0" (default) 2022-04-02T03:09:15.182 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:09:15.180519 I | op-k8sutil: ROOK_CSI_SNAPSHOTTER_IMAGE="k8s.gcr.io/sig-storage/csi-snapshotter:v5.0.1" (default) 2022-04-02T03:09:15.182 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:09:15.180526 I | op-k8sutil: ROOK_CSI_KUBELET_DIR_PATH="/var/lib/kubelet" (default) 2022-04-02T03:09:15.182 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:09:15.180548 I | op-k8sutil: CSI_VOLUME_REPLICATION_IMAGE="quay.io/csiaddons/volumereplication-operator:v0.3.0" (default) 2022-04-02T03:09:15.182 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:09:15.180560 I | op-k8sutil: ROOK_CSIADDONS_IMAGE="quay.io/csiaddons/k8s-sidecar:v0.2.1" (default) 2022-04-02T03:09:15.182 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:09:15.180566 I | op-k8sutil: ROOK_CSI_CEPHFS_POD_LABELS="" (default) 2022-04-02T03:09:15.183 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:09:15.180573 I | op-k8sutil: ROOK_CSI_RBD_POD_LABELS="" (default) 2022-04-02T03:09:15.183 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:09:15.180581 I | ceph-csi: detecting the ceph csi image version for image "quay.io/cephcsi/cephcsi:v3.5.1" 2022-04-02T03:09:15.183 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:09:15.180651 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-04-02T03:09:15.183 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:09:15.180664 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default) 2022-04-02T03:09:15.331 INFO:teuthology.orchestra.run.smithi029.stdout:deployment.apps/rook-ceph-tools created 2022-04-02T03:09:15.336 INFO:tasks.rook:Waiting for tools container to start 2022-04-02T03:09:15.336 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-04-02T03:09:15.484 INFO:teuthology.orchestra.run.smithi029.stdout:NAME READY STATUS RESTARTS AGE 2022-04-02T03:09:15.484 INFO:teuthology.orchestra.run.smithi029.stdout:rook-ceph-tools-d6d7c985c-2ddzt 0/1 Pending 0 0s 2022-04-02T03:09:20.489 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-04-02T03:09:20.562 INFO:teuthology.orchestra.run.smithi029.stdout:NAME READY STATUS RESTARTS AGE 2022-04-02T03:09:20.562 INFO:teuthology.orchestra.run.smithi029.stdout:rook-ceph-tools-d6d7c985c-2ddzt 0/1 ContainerCreating 0 5s 2022-04-02T03:09:25.565 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-04-02T03:09:25.628 INFO:teuthology.orchestra.run.smithi029.stdout:NAME READY STATUS RESTARTS AGE 2022-04-02T03:09:25.629 INFO:teuthology.orchestra.run.smithi029.stdout:rook-ceph-tools-d6d7c985c-2ddzt 0/1 ContainerCreating 0 10s 2022-04-02T03:09:30.631 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-04-02T03:09:30.996 INFO:teuthology.orchestra.run.smithi029.stdout:NAME READY STATUS RESTARTS AGE 2022-04-02T03:09:30.997 INFO:teuthology.orchestra.run.smithi029.stdout:rook-ceph-tools-d6d7c985c-2ddzt 0/1 ContainerCreating 0 15s 2022-04-02T03:09:35.999 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-04-02T03:09:36.074 INFO:teuthology.orchestra.run.smithi029.stdout:NAME READY STATUS RESTARTS AGE 2022-04-02T03:09:36.074 INFO:teuthology.orchestra.run.smithi029.stdout:rook-ceph-tools-d6d7c985c-2ddzt 0/1 ContainerCreating 0 21s 2022-04-02T03:09:41.078 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-04-02T03:09:41.142 INFO:teuthology.orchestra.run.smithi029.stdout:NAME READY STATUS RESTARTS AGE 2022-04-02T03:09:41.142 INFO:teuthology.orchestra.run.smithi029.stdout:rook-ceph-tools-d6d7c985c-2ddzt 0/1 ContainerCreating 0 26s 2022-04-02T03:09:46.145 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-04-02T03:09:46.216 INFO:teuthology.orchestra.run.smithi029.stdout:NAME READY STATUS RESTARTS AGE 2022-04-02T03:09:46.217 INFO:teuthology.orchestra.run.smithi029.stdout:rook-ceph-tools-d6d7c985c-2ddzt 0/1 ContainerCreating 0 31s 2022-04-02T03:09:50.748 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:09:50.747092 I | ceph-spec: detected ceph image version: "17.0.0-11449 quincy" 2022-04-02T03:09:50.749 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:09:50.747114 I | ceph-cluster-controller: validating ceph version from provided image 2022-04-02T03:09:50.778 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:09:50.772803 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-11449 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a" 2022-04-02T03:09:50.901 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:09:50.899482 E | ceph-spec: failed to update cluster condition to {Type:Progressing Status:True Reason:ClusterProgressing Message:Configuring the Ceph cluster LastHeartbeatTime:2022-04-02 03:09:50.8744465 +0000 UTC m=+48.662343202 LastTransitionTime:2022-04-02 03:09:50.874446418 +0000 UTC m=+48.662343139}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-04-02T03:09:50.979 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:09:50.977642 I | op-mon: start running mons 2022-04-02T03:09:51.041 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:09:51.039008 I | op-mon: creating mon secrets for a new cluster 2022-04-02T03:09:51.063 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:09:51.062066 I | op-mon: existing maxMonID not found or failed to load. configmaps "rook-ceph-mon-endpoints" not found 2022-04-02T03:09:51.066 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:09:51.065401 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":[]}] data: mapping:{"node":{}} maxMonId:-1] 2022-04-02T03:09:51.218 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-04-02T03:09:51.282 INFO:teuthology.orchestra.run.smithi029.stdout:NAME READY STATUS RESTARTS AGE 2022-04-02T03:09:51.282 INFO:teuthology.orchestra.run.smithi029.stdout:rook-ceph-tools-d6d7c985c-2ddzt 0/1 ContainerCreating 0 36s 2022-04-02T03:09:51.747 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:09:51.739576 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-04-02T03:09:51.748 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:09:51.739833 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-04-02T03:09:53.341 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:09:53.339173 I | op-mon: targeting the mon count 3 2022-04-02T03:09:53.357 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:09:53.355207 I | op-mon: created canary deployment rook-ceph-mon-a-canary 2022-04-02T03:09:53.370 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:09:53.367924 I | op-mon: created canary deployment rook-ceph-mon-b-canary 2022-04-02T03:09:53.438 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:09:53.436715 I | op-mon: created canary deployment rook-ceph-mon-c-canary 2022-04-02T03:09:54.141 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:09:54.139175 I | op-mon: canary monitor deployment rook-ceph-mon-a-canary scheduled to smithi104 2022-04-02T03:09:54.141 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:09:54.139213 I | op-mon: mon a assigned to node smithi104 2022-04-02T03:09:54.341 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:09:54.339615 I | op-mon: canary monitor deployment rook-ceph-mon-b-canary scheduled to smithi142 2022-04-02T03:09:54.342 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:09:54.339634 I | op-mon: mon b assigned to node smithi142 2022-04-02T03:09:54.540 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:09:54.538086 I | op-mon: canary monitor deployment rook-ceph-mon-c-canary scheduled to smithi029 2022-04-02T03:09:54.540 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:09:54.538119 I | op-mon: mon c assigned to node smithi029 2022-04-02T03:09:54.545 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:09:54.543113 I | op-mon: cleaning up canary monitor deployment "rook-ceph-mon-a-canary" 2022-04-02T03:09:54.549 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:09:54.547950 I | op-mon: cleaning up canary monitor deployment "rook-ceph-mon-b-canary" 2022-04-02T03:09:54.707 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:09:54.705034 I | op-mon: cleaning up canary monitor deployment "rook-ceph-mon-c-canary" 2022-04-02T03:09:54.870 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:09:54.868159 I | op-mon: creating mon a 2022-04-02T03:09:55.099 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:09:55.097677 I | op-mon: mon "a" ip is 10.102.133.199 2022-04-02T03:09:55.607 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:09:55.605056 I | op-mon: monitor endpoints changed, updating the bootstrap peer token 2022-04-02T03:09:55.607 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:09:55.605087 I | op-mon: monitor endpoints changed, updating the bootstrap peer token 2022-04-02T03:09:55.610 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:09:55.605355 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.102.133.199:6789"]}] data:a=10.102.133.199:6789 mapping:{"node":{"a":{"Name":"smithi104","Hostname":"smithi104","Address":"172.21.15.104"},"b":{"Name":"smithi142","Hostname":"smithi142","Address":"172.21.15.142"},"c":{"Name":"smithi029","Hostname":"smithi029","Address":"172.21.15.29"}}} maxMonId:-1] 2022-04-02T03:09:55.753 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:09:55.750837 I | op-mon: parsing mon endpoints: a=10.102.133.199:6789 2022-04-02T03:09:55.753 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:09:55.750875 I | op-mon: updating obsolete maxMonID -1 to actual value 0 2022-04-02T03:09:55.753 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:09:55.750935 I | op-k8sutil: ROOK_OBC_WATCH_OPERATOR_NAMESPACE="true" (configmap) 2022-04-02T03:09:55.754 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:09:55.750952 I | op-bucket-prov: ceph bucket provisioner launched watching for provisioner "rook-ceph.ceph.rook.io/bucket" 2022-04-02T03:09:55.757 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:09:55.753269 I | op-bucket-prov: successfully reconciled bucket provisioner 2022-04-02T03:09:55.757 INFO:tasks.rook.operator.smithi029.stdout:I0402 03:09:55.753381 1 manager.go:135] objectbucket.io/provisioner-manager "msg"="starting provisioner" "name"="rook-ceph.ceph.rook.io/bucket" 2022-04-02T03:09:56.285 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-04-02T03:09:56.363 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:09:56.359515 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-04-02T03:09:56.489 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:09:56.359838 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-04-02T03:09:56.490 INFO:teuthology.orchestra.run.smithi029.stdout:NAME READY STATUS RESTARTS AGE 2022-04-02T03:09:56.491 INFO:teuthology.orchestra.run.smithi029.stdout:rook-ceph-tools-d6d7c985c-2ddzt 0/1 ContainerCreating 0 41s 2022-04-02T03:09:56.762 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:09:56.759825 I | op-mon: 0 of 1 expected mons are ready. creating or updating deployments without checking quorum in attempt to achieve a healthy mon cluster 2022-04-02T03:09:56.939 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:09:56.937676 I | op-mon: updating maxMonID from -1 to 0 after committing mon "a" 2022-04-02T03:09:57.811 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:09:57.809322 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.102.133.199:6789"]}] data:a=10.102.133.199:6789 mapping:{"node":{"a":{"Name":"smithi104","Hostname":"smithi104","Address":"172.21.15.104"},"b":{"Name":"smithi142","Hostname":"smithi142","Address":"172.21.15.142"},"c":{"Name":"smithi029","Hostname":"smithi029","Address":"172.21.15.29"}}} maxMonId:0] 2022-04-02T03:09:57.811 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:09:57.809355 I | op-mon: waiting for mon quorum with [a] 2022-04-02T03:09:57.942 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:09:57.939994 I | op-mon: mons running: [a] 2022-04-02T03:10:01.492 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-04-02T03:10:01.561 INFO:teuthology.orchestra.run.smithi029.stdout:NAME READY STATUS RESTARTS AGE 2022-04-02T03:10:01.561 INFO:teuthology.orchestra.run.smithi029.stdout:rook-ceph-tools-d6d7c985c-2ddzt 0/1 ContainerCreating 0 46s 2022-04-02T03:10:06.564 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-04-02T03:10:06.638 INFO:teuthology.orchestra.run.smithi029.stdout:NAME READY STATUS RESTARTS AGE 2022-04-02T03:10:06.638 INFO:teuthology.orchestra.run.smithi029.stdout:rook-ceph-tools-d6d7c985c-2ddzt 0/1 ContainerCreating 0 51s 2022-04-02T03:10:11.641 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-04-02T03:10:11.793 INFO:teuthology.orchestra.run.smithi029.stdout:NAME READY STATUS RESTARTS AGE 2022-04-02T03:10:11.794 INFO:teuthology.orchestra.run.smithi029.stdout:rook-ceph-tools-d6d7c985c-2ddzt 0/1 ContainerCreating 0 56s 2022-04-02T03:10:16.796 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-04-02T03:10:16.864 INFO:teuthology.orchestra.run.smithi029.stdout:NAME READY STATUS RESTARTS AGE 2022-04-02T03:10:16.865 INFO:teuthology.orchestra.run.smithi029.stdout:rook-ceph-tools-d6d7c985c-2ddzt 0/1 ContainerCreating 0 61s 2022-04-02T03:10:18.678 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:10:18.676820 I | op-mon: mons running: [a] 2022-04-02T03:10:21.867 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-04-02T03:10:22.418 INFO:teuthology.orchestra.run.smithi029.stdout:NAME READY STATUS RESTARTS AGE 2022-04-02T03:10:22.418 INFO:teuthology.orchestra.run.smithi029.stdout:rook-ceph-tools-d6d7c985c-2ddzt 0/1 ContainerCreating 0 67s 2022-04-02T03:10:27.422 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-04-02T03:10:27.483 INFO:teuthology.orchestra.run.smithi029.stdout:NAME READY STATUS RESTARTS AGE 2022-04-02T03:10:27.483 INFO:teuthology.orchestra.run.smithi029.stdout:rook-ceph-tools-d6d7c985c-2ddzt 1/1 Running 0 72s 2022-04-02T03:10:27.485 INFO:tasks.rook:Waiting for mgr/rook orchestrator to be available 2022-04-02T03:10:27.485 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph orch status -f json 2022-04-02T03:10:38.839 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:10:38.837909 I | op-mon: mons running: [a] 2022-04-02T03:10:42.411 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:10:42.408035 I | ceph-csi: Detected ceph CSI image version: "v3.5.1" 2022-04-02T03:10:42.411 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:10:42.408075 I | op-k8sutil: CSI_FORCE_CEPHFS_KERNEL_CLIENT="true" (configmap) 2022-04-02T03:10:42.412 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:10:42.408087 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default) 2022-04-02T03:10:42.412 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:10:42.408094 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default) 2022-04-02T03:10:42.412 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:10:42.408103 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default) 2022-04-02T03:10:42.412 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:10:42.408111 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default) 2022-04-02T03:10:42.413 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:10:42.408125 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default) 2022-04-02T03:10:42.413 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:10:42.408138 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default) 2022-04-02T03:10:42.414 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:10:42.408150 I | op-k8sutil: CSIADDONS_PORT="9070" (default) 2022-04-02T03:10:42.414 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:10:42.408161 I | op-k8sutil: CSIADDONS_PORT="9070" (default) 2022-04-02T03:10:42.414 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:10:42.408174 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default) 2022-04-02T03:10:42.414 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:10:42.408179 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default) 2022-04-02T03:10:42.414 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:10:42.408187 I | op-k8sutil: CSI_PLUGIN_PRIORITY_CLASSNAME="" (default) 2022-04-02T03:10:42.415 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:10:42.408200 I | op-k8sutil: CSI_PROVISIONER_PRIORITY_CLASSNAME="" (default) 2022-04-02T03:10:42.416 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:10:42.408211 I | op-k8sutil: CSI_ENABLE_OMAP_GENERATOR="false" (default) 2022-04-02T03:10:42.416 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:10:42.408226 I | op-k8sutil: CSI_ENABLE_RBD_SNAPSHOTTER="true" (configmap) 2022-04-02T03:10:42.416 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:10:42.408238 I | op-k8sutil: CSI_ENABLE_CEPHFS_SNAPSHOTTER="true" (configmap) 2022-04-02T03:10:42.417 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:10:42.408249 I | op-k8sutil: CSI_ENABLE_VOLUME_REPLICATION="false" (configmap) 2022-04-02T03:10:42.417 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:10:42.408263 I | op-k8sutil: CSI_ENABLE_CSIADDONS="false" (configmap) 2022-04-02T03:10:42.417 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:10:42.408274 I | op-k8sutil: CSI_CEPHFS_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-04-02T03:10:42.417 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:10:42.408285 I | op-k8sutil: CSI_RBD_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-04-02T03:10:42.417 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:10:42.408297 I | op-k8sutil: CSI_PLUGIN_ENABLE_SELINUX_HOST_MOUNT="false" (configmap) 2022-04-02T03:10:42.418 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:10:42.408305 I | ceph-csi: Kubernetes version is 1.23 2022-04-02T03:10:42.418 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:10:42.408314 I | op-k8sutil: ROOK_CSI_RESIZER_IMAGE="k8s.gcr.io/sig-storage/csi-resizer:v1.4.0" (default) 2022-04-02T03:10:42.418 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:10:42.408396 I | op-k8sutil: CSI_LOG_LEVEL="" (default) 2022-04-02T03:10:42.427 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:10:42.425958 I | op-k8sutil: CSI_PROVISIONER_REPLICAS="2" (configmap) 2022-04-02T03:10:42.437 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:10:42.435790 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-04-02T03:10:42.437 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:10:42.435813 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default) 2022-04-02T03:10:42.437 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:10:42.435820 I | op-k8sutil: CSI_PLUGIN_TOLERATIONS="" (default) 2022-04-02T03:10:42.437 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:10:42.435827 I | op-k8sutil: CSI_PLUGIN_NODE_AFFINITY="" (default) 2022-04-02T03:10:42.438 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:10:42.435832 I | op-k8sutil: CSI_RBD_PLUGIN_TOLERATIONS="" (default) 2022-04-02T03:10:42.438 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:10:42.435838 I | op-k8sutil: CSI_RBD_PLUGIN_NODE_AFFINITY="" (default) 2022-04-02T03:10:42.438 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:10:42.435846 I | op-k8sutil: CSI_RBD_PLUGIN_RESOURCE="" (default) 2022-04-02T03:10:42.479 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:10:42.475354 I | op-k8sutil: CSI_RBD_PROVISIONER_TOLERATIONS="" (default) 2022-04-02T03:10:42.479 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:10:42.475412 I | op-k8sutil: CSI_RBD_PROVISIONER_NODE_AFFINITY="" (default) 2022-04-02T03:10:42.479 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:10:42.475426 I | op-k8sutil: CSI_RBD_PROVISIONER_RESOURCE="" (default) 2022-04-02T03:10:42.588 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:10:42.586556 I | ceph-csi: successfully started CSI Ceph RBD driver 2022-04-02T03:10:42.722 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:10:42.721354 I | op-k8sutil: CSI_CEPHFS_PLUGIN_TOLERATIONS="" (default) 2022-04-02T03:10:42.723 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:10:42.721391 I | op-k8sutil: CSI_CEPHFS_PLUGIN_NODE_AFFINITY="" (default) 2022-04-02T03:10:42.724 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:10:42.721406 I | op-k8sutil: CSI_CEPHFS_PLUGIN_RESOURCE="" (default) 2022-04-02T03:10:42.765 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:10:42.764291 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_TOLERATIONS="" (default) 2022-04-02T03:10:42.766 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:10:42.764316 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_NODE_AFFINITY="" (default) 2022-04-02T03:10:42.766 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:10:42.764323 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_RESOURCE="" (default) 2022-04-02T03:10:42.980 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:10:42.978969 I | ceph-csi: successfully started CSI CephFS driver 2022-04-02T03:10:43.773 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:10:43.772767 I | op-k8sutil: CSI_RBD_FSGROUPPOLICY="ReadWriteOnceWithFSType" (configmap) 2022-04-02T03:10:44.272 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:10:44.271129 I | ceph-csi: CSIDriver object created for driver "rook-ceph.rbd.csi.ceph.com" 2022-04-02T03:10:44.273 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:10:44.271147 I | op-k8sutil: CSI_CEPHFS_FSGROUPPOLICY="ReadWriteOnceWithFSType" (configmap) 2022-04-02T03:10:44.465 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:10:44.464462 I | ceph-csi: CSIDriver object created for driver "rook-ceph.cephfs.csi.ceph.com" 2022-04-02T03:10:59.016 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:10:59.014856 I | op-mon: mon a is not yet running 2022-04-02T03:10:59.016 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:10:59.014889 I | op-mon: mons running: [] 2022-04-02T03:11:19.187 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:11:19.186677 I | op-mon: mon a is not yet running 2022-04-02T03:11:19.188 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:11:19.186704 I | op-mon: mons running: [] 2022-04-02T03:11:39.462 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:11:39.462395 I | op-mon: mons running: [a] 2022-04-02T03:11:40.386 INFO:teuthology.orchestra.run.smithi029.stdout: 2022-04-02T03:11:40.387 INFO:teuthology.orchestra.run.smithi029.stderr:no valid command found; 10 closest matches: 2022-04-02T03:11:40.387 INFO:teuthology.orchestra.run.smithi029.stderr:pg stat 2022-04-02T03:11:40.387 INFO:teuthology.orchestra.run.smithi029.stderr:pg getmap 2022-04-02T03:11:40.388 INFO:teuthology.orchestra.run.smithi029.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-04-02T03:11:40.388 INFO:teuthology.orchestra.run.smithi029.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-04-02T03:11:40.388 INFO:teuthology.orchestra.run.smithi029.stderr:pg dump_pools_json 2022-04-02T03:11:40.388 INFO:teuthology.orchestra.run.smithi029.stderr:pg ls-by-pool [...] 2022-04-02T03:11:40.389 INFO:teuthology.orchestra.run.smithi029.stderr:pg ls-by-primary [] [...] 2022-04-02T03:11:40.389 INFO:teuthology.orchestra.run.smithi029.stderr:pg ls-by-osd [] [...] 2022-04-02T03:11:40.389 INFO:teuthology.orchestra.run.smithi029.stderr:pg ls [] [...] 2022-04-02T03:11:40.390 INFO:teuthology.orchestra.run.smithi029.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-04-02T03:11:40.390 INFO:teuthology.orchestra.run.smithi029.stderr:Error EINVAL: invalid command 2022-04-02T03:11:40.391 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-04-02T03:11:40.393 INFO:teuthology.orchestra.run.smithi029.stderr:command terminated with exit code 22 2022-04-02T03:11:40.724 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:11:40.724525 I | op-mon: Monitors in quorum: [a] 2022-04-02T03:11:40.725 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:11:40.724564 I | op-mon: mons created: 1 2022-04-02T03:11:40.957 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:11:40.957217 I | op-mon: waiting for mon quorum with [a] 2022-04-02T03:11:41.093 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:11:41.093548 I | op-mon: mons running: [a] 2022-04-02T03:11:41.317 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:11:41.317631 I | op-mon: Monitors in quorum: [a] 2022-04-02T03:11:41.318 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:11:41.317668 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-04-02T03:11:41.597 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:11:41.596907 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-04-02T03:11:41.597 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:11:41.596937 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-04-02T03:11:41.867 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:11:41.867722 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-04-02T03:11:41.868 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:11:41.867751 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-04-02T03:11:42.141 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:11:42.141265 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-04-02T03:11:42.141 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:11:42.141305 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-04-02T03:11:42.394 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:11:42.394621 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-04-02T03:11:42.394 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:11:42.394651 I | op-config: deleting "log file" option from the mon configuration database 2022-04-02T03:11:42.594 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:11:42.594376 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-04-02T03:11:42.594 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:11:42.594396 I | op-mon: creating mon b 2022-04-02T03:11:46.518 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:11:46.518676 I | op-mon: mon "a" ip is 10.102.133.199 2022-04-02T03:11:47.026 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:11:47.026007 I | op-mon: mon "b" ip is 10.107.147.59 2022-04-02T03:11:48.498 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:11:48.498648 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.102.133.199:6789","10.107.147.59:6789"]}] data:a=10.102.133.199:6789,b=10.107.147.59:6789 mapping:{"node":{"a":{"Name":"smithi104","Hostname":"smithi104","Address":"172.21.15.104"},"b":{"Name":"smithi142","Hostname":"smithi142","Address":"172.21.15.142"},"c":{"Name":"smithi029","Hostname":"smithi029","Address":"172.21.15.29"}}} maxMonId:0] 2022-04-02T03:11:48.769 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:11:48.769931 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-04-02T03:11:48.790 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:11:48.790177 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-04-02T03:11:49.533 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:11:49.533654 I | op-mon: 1 of 2 expected mon deployments exist. creating new deployment(s). 2022-04-02T03:11:49.541 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:11:49.539952 I | op-mon: deployment for mon rook-ceph-mon-a already exists. updating if needed 2022-04-02T03:11:49.547 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:11:49.547645 I | op-k8sutil: deployment "rook-ceph-mon-a" did not change, nothing to update 2022-04-02T03:11:49.977 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:11:49.976887 I | op-mon: updating maxMonID from 0 to 1 after committing mon "b" 2022-04-02T03:11:50.395 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph orch status -f json 2022-04-02T03:11:52.893 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:11:52.893968 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.102.133.199:6789","10.107.147.59:6789"]}] data:a=10.102.133.199:6789,b=10.107.147.59:6789 mapping:{"node":{"a":{"Name":"smithi104","Hostname":"smithi104","Address":"172.21.15.104"},"b":{"Name":"smithi142","Hostname":"smithi142","Address":"172.21.15.142"},"c":{"Name":"smithi029","Hostname":"smithi029","Address":"172.21.15.29"}}} maxMonId:1] 2022-04-02T03:11:52.894 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:11:52.893992 I | op-mon: waiting for mon quorum with [a b] 2022-04-02T03:11:53.323 INFO:teuthology.orchestra.run.smithi029.stdout: 2022-04-02T03:11:53.324 INFO:teuthology.orchestra.run.smithi029.stderr:no valid command found; 10 closest matches: 2022-04-02T03:11:53.324 INFO:teuthology.orchestra.run.smithi029.stderr:pg stat 2022-04-02T03:11:53.324 INFO:teuthology.orchestra.run.smithi029.stderr:pg getmap 2022-04-02T03:11:53.324 INFO:teuthology.orchestra.run.smithi029.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-04-02T03:11:53.325 INFO:teuthology.orchestra.run.smithi029.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-04-02T03:11:53.325 INFO:teuthology.orchestra.run.smithi029.stderr:pg dump_pools_json 2022-04-02T03:11:53.325 INFO:teuthology.orchestra.run.smithi029.stderr:pg ls-by-pool [...] 2022-04-02T03:11:53.325 INFO:teuthology.orchestra.run.smithi029.stderr:pg ls-by-primary [] [...] 2022-04-02T03:11:53.325 INFO:teuthology.orchestra.run.smithi029.stderr:pg ls-by-osd [] [...] 2022-04-02T03:11:53.326 INFO:teuthology.orchestra.run.smithi029.stderr:pg ls [] [...] 2022-04-02T03:11:53.326 INFO:teuthology.orchestra.run.smithi029.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-04-02T03:11:53.326 INFO:teuthology.orchestra.run.smithi029.stderr:Error EINVAL: invalid command 2022-04-02T03:11:53.327 INFO:teuthology.orchestra.run.smithi029.stderr:command terminated with exit code 22 2022-04-02T03:11:53.331 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-04-02T03:11:54.331 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:11:54.331351 I | op-mon: mon b is not yet running 2022-04-02T03:11:54.331 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:11:54.331388 I | op-mon: mons running: [a] 2022-04-02T03:11:54.564 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:11:54.563668 I | op-mon: Monitors in quorum: [a] 2022-04-02T03:11:54.564 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:11:54.563693 I | op-mon: mons created: 2 2022-04-02T03:11:54.789 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:11:54.789243 I | op-mon: waiting for mon quorum with [a b] 2022-04-02T03:11:54.991 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:11:54.991463 I | op-mon: mon b is not yet running 2022-04-02T03:11:54.991 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:11:54.991498 I | op-mon: mons running: [a] 2022-04-02T03:12:00.014 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:12:00.014912 I | op-mon: mon b is not yet running 2022-04-02T03:12:00.015 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:12:00.014960 I | op-mon: mons running: [a] 2022-04-02T03:12:03.333 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph orch status -f json 2022-04-02T03:12:03.817 INFO:teuthology.orchestra.run.smithi029.stdout: 2022-04-02T03:12:03.818 INFO:teuthology.orchestra.run.smithi029.stderr:no valid command found; 10 closest matches: 2022-04-02T03:12:03.818 INFO:teuthology.orchestra.run.smithi029.stderr:pg stat 2022-04-02T03:12:03.818 INFO:teuthology.orchestra.run.smithi029.stderr:pg getmap 2022-04-02T03:12:03.818 INFO:teuthology.orchestra.run.smithi029.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-04-02T03:12:03.819 INFO:teuthology.orchestra.run.smithi029.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-04-02T03:12:03.819 INFO:teuthology.orchestra.run.smithi029.stderr:pg dump_pools_json 2022-04-02T03:12:03.819 INFO:teuthology.orchestra.run.smithi029.stderr:pg ls-by-pool [...] 2022-04-02T03:12:03.819 INFO:teuthology.orchestra.run.smithi029.stderr:pg ls-by-primary [] [...] 2022-04-02T03:12:03.820 INFO:teuthology.orchestra.run.smithi029.stderr:pg ls-by-osd [] [...] 2022-04-02T03:12:03.820 INFO:teuthology.orchestra.run.smithi029.stderr:pg ls [] [...] 2022-04-02T03:12:03.820 INFO:teuthology.orchestra.run.smithi029.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-04-02T03:12:03.820 INFO:teuthology.orchestra.run.smithi029.stderr:Error EINVAL: invalid command 2022-04-02T03:12:03.821 INFO:teuthology.orchestra.run.smithi029.stderr:command terminated with exit code 22 2022-04-02T03:12:03.823 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-04-02T03:12:05.047 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:12:05.045229 I | op-mon: mon b is not yet running 2022-04-02T03:12:05.047 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:12:05.045254 I | op-mon: mons running: [a] 2022-04-02T03:12:10.062 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:12:10.063108 I | op-mon: mons running: [a b] 2022-04-02T03:12:11.698 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:12:11.698776 I | op-mon: Monitors in quorum: [a b] 2022-04-02T03:12:11.698 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:12:11.698816 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-04-02T03:12:11.930 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:12:11.931138 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-04-02T03:12:11.931 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:12:11.931168 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-04-02T03:12:12.168 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:12:12.168610 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-04-02T03:12:12.168 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:12:12.168643 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-04-02T03:12:12.392 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:12:12.392510 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-04-02T03:12:12.392 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:12:12.392556 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-04-02T03:12:12.611 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:12:12.611750 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-04-02T03:12:12.611 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:12:12.611791 I | op-config: deleting "log file" option from the mon configuration database 2022-04-02T03:12:12.825 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:12:12.825807 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-04-02T03:12:12.826 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:12:12.825846 I | op-mon: creating mon c 2022-04-02T03:12:12.957 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:12:12.957901 I | op-mon: mon "a" ip is 10.102.133.199 2022-04-02T03:12:13.102 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:12:13.103402 I | op-mon: mon "b" ip is 10.107.147.59 2022-04-02T03:12:13.137 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:12:13.137932 I | op-mon: mon "c" ip is 10.96.208.36 2022-04-02T03:12:13.235 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:12:13.235740 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.102.133.199:6789","10.107.147.59:6789","10.96.208.36:6789"]}] data:b=10.107.147.59:6789,c=10.96.208.36:6789,a=10.102.133.199:6789 mapping:{"node":{"a":{"Name":"smithi104","Hostname":"smithi104","Address":"172.21.15.104"},"b":{"Name":"smithi142","Hostname":"smithi142","Address":"172.21.15.142"},"c":{"Name":"smithi029","Hostname":"smithi029","Address":"172.21.15.29"}}} maxMonId:1] 2022-04-02T03:12:13.825 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph orch status -f json 2022-04-02T03:12:13.829 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:12:13.829871 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-04-02T03:12:13.834 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:12:13.830184 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-04-02T03:12:14.241 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:12:14.242435 I | op-mon: 2 of 3 expected mon deployments exist. creating new deployment(s). 2022-04-02T03:12:14.272 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:12:14.254414 I | op-mon: deployment for mon rook-ceph-mon-a already exists. updating if needed 2022-04-02T03:12:14.272 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:12:14.268057 I | op-k8sutil: deployment "rook-ceph-mon-a" did not change, nothing to update 2022-04-02T03:12:14.272 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:12:14.273069 I | op-mon: deployment for mon rook-ceph-mon-b already exists. updating if needed 2022-04-02T03:12:14.280 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:12:14.281795 I | op-k8sutil: deployment "rook-ceph-mon-b" did not change, nothing to update 2022-04-02T03:12:14.309 INFO:teuthology.orchestra.run.smithi029.stdout: 2022-04-02T03:12:14.309 INFO:teuthology.orchestra.run.smithi029.stderr:no valid command found; 10 closest matches: 2022-04-02T03:12:14.310 INFO:teuthology.orchestra.run.smithi029.stderr:pg stat 2022-04-02T03:12:14.310 INFO:teuthology.orchestra.run.smithi029.stderr:pg getmap 2022-04-02T03:12:14.310 INFO:teuthology.orchestra.run.smithi029.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-04-02T03:12:14.310 INFO:teuthology.orchestra.run.smithi029.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-04-02T03:12:14.310 INFO:teuthology.orchestra.run.smithi029.stderr:pg dump_pools_json 2022-04-02T03:12:14.311 INFO:teuthology.orchestra.run.smithi029.stderr:pg ls-by-pool [...] 2022-04-02T03:12:14.311 INFO:teuthology.orchestra.run.smithi029.stderr:pg ls-by-primary [] [...] 2022-04-02T03:12:14.311 INFO:teuthology.orchestra.run.smithi029.stderr:pg ls-by-osd [] [...] 2022-04-02T03:12:14.311 INFO:teuthology.orchestra.run.smithi029.stderr:pg ls [] [...] 2022-04-02T03:12:14.311 INFO:teuthology.orchestra.run.smithi029.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-04-02T03:12:14.312 INFO:teuthology.orchestra.run.smithi029.stderr:Error EINVAL: invalid command 2022-04-02T03:12:14.314 INFO:teuthology.orchestra.run.smithi029.stderr:command terminated with exit code 22 2022-04-02T03:12:14.316 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-04-02T03:12:14.488 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:12:14.488648 I | op-mon: updating maxMonID from 1 to 2 after committing mon "c" 2022-04-02T03:12:15.386 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:12:15.386994 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.102.133.199:6789","10.107.147.59:6789","10.96.208.36:6789"]}] data:a=10.102.133.199:6789,b=10.107.147.59:6789,c=10.96.208.36:6789 mapping:{"node":{"a":{"Name":"smithi104","Hostname":"smithi104","Address":"172.21.15.104"},"b":{"Name":"smithi142","Hostname":"smithi142","Address":"172.21.15.142"},"c":{"Name":"smithi029","Hostname":"smithi029","Address":"172.21.15.29"}}} maxMonId:2] 2022-04-02T03:12:15.387 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:12:15.387033 I | op-mon: waiting for mon quorum with [a b c] 2022-04-02T03:12:15.834 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:12:15.834635 I | op-mon: mon c is not yet running 2022-04-02T03:12:15.838 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:12:15.834676 I | op-mon: mons running: [a b] 2022-04-02T03:12:16.063 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:12:16.063566 I | op-mon: Monitors in quorum: [a b] 2022-04-02T03:12:16.063 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:12:16.063596 I | op-mon: mons created: 3 2022-04-02T03:12:16.280 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:12:16.280759 I | op-mon: waiting for mon quorum with [a b c] 2022-04-02T03:12:16.435 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:12:16.435898 I | op-mon: mon c is not yet running 2022-04-02T03:12:16.436 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:12:16.435929 I | op-mon: mons running: [a b] 2022-04-02T03:12:21.517 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:12:21.518719 I | op-mon: mon c is not yet running 2022-04-02T03:12:21.518 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:12:21.518751 I | op-mon: mons running: [a b] 2022-04-02T03:12:24.318 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph orch status -f json 2022-04-02T03:12:24.799 INFO:teuthology.orchestra.run.smithi029.stdout: 2022-04-02T03:12:24.799 INFO:teuthology.orchestra.run.smithi029.stderr:no valid command found; 10 closest matches: 2022-04-02T03:12:24.800 INFO:teuthology.orchestra.run.smithi029.stderr:pg stat 2022-04-02T03:12:24.800 INFO:teuthology.orchestra.run.smithi029.stderr:pg getmap 2022-04-02T03:12:24.800 INFO:teuthology.orchestra.run.smithi029.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-04-02T03:12:24.800 INFO:teuthology.orchestra.run.smithi029.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-04-02T03:12:24.800 INFO:teuthology.orchestra.run.smithi029.stderr:pg dump_pools_json 2022-04-02T03:12:24.801 INFO:teuthology.orchestra.run.smithi029.stderr:pg ls-by-pool [...] 2022-04-02T03:12:24.801 INFO:teuthology.orchestra.run.smithi029.stderr:pg ls-by-primary [] [...] 2022-04-02T03:12:24.801 INFO:teuthology.orchestra.run.smithi029.stderr:pg ls-by-osd [] [...] 2022-04-02T03:12:24.801 INFO:teuthology.orchestra.run.smithi029.stderr:pg ls [] [...] 2022-04-02T03:12:24.801 INFO:teuthology.orchestra.run.smithi029.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-04-02T03:12:24.802 INFO:teuthology.orchestra.run.smithi029.stderr:Error EINVAL: invalid command 2022-04-02T03:12:24.804 INFO:teuthology.orchestra.run.smithi029.stderr:command terminated with exit code 22 2022-04-02T03:12:24.805 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-04-02T03:12:26.555 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:12:26.556497 I | op-mon: mon c is not yet running 2022-04-02T03:12:26.556 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:12:26.556526 I | op-mon: mons running: [a b] 2022-04-02T03:12:31.577 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:12:31.578704 I | op-mon: mon c is not yet running 2022-04-02T03:12:31.578 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:12:31.578740 I | op-mon: mons running: [a b] 2022-04-02T03:12:34.807 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph orch status -f json 2022-04-02T03:12:35.295 INFO:teuthology.orchestra.run.smithi029.stdout: 2022-04-02T03:12:35.296 INFO:teuthology.orchestra.run.smithi029.stderr:no valid command found; 10 closest matches: 2022-04-02T03:12:35.296 INFO:teuthology.orchestra.run.smithi029.stderr:pg stat 2022-04-02T03:12:35.297 INFO:teuthology.orchestra.run.smithi029.stderr:pg getmap 2022-04-02T03:12:35.297 INFO:teuthology.orchestra.run.smithi029.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-04-02T03:12:35.297 INFO:teuthology.orchestra.run.smithi029.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-04-02T03:12:35.297 INFO:teuthology.orchestra.run.smithi029.stderr:pg dump_pools_json 2022-04-02T03:12:35.298 INFO:teuthology.orchestra.run.smithi029.stderr:pg ls-by-pool [...] 2022-04-02T03:12:35.298 INFO:teuthology.orchestra.run.smithi029.stderr:pg ls-by-primary [] [...] 2022-04-02T03:12:35.298 INFO:teuthology.orchestra.run.smithi029.stderr:pg ls-by-osd [] [...] 2022-04-02T03:12:35.299 INFO:teuthology.orchestra.run.smithi029.stderr:pg ls [] [...] 2022-04-02T03:12:35.299 INFO:teuthology.orchestra.run.smithi029.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-04-02T03:12:35.299 INFO:teuthology.orchestra.run.smithi029.stderr:Error EINVAL: invalid command 2022-04-02T03:12:35.300 INFO:teuthology.orchestra.run.smithi029.stderr:command terminated with exit code 22 2022-04-02T03:12:35.302 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-04-02T03:12:36.615 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:12:36.615593 I | op-mon: mon c is not yet running 2022-04-02T03:12:36.615 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:12:36.615622 I | op-mon: mons running: [a b] 2022-04-02T03:12:41.990 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:12:41.991158 I | op-mon: mon c is not yet running 2022-04-02T03:12:41.990 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:12:41.991193 I | op-mon: mons running: [a b] 2022-04-02T03:12:45.304 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph orch status -f json 2022-04-02T03:12:45.793 INFO:teuthology.orchestra.run.smithi029.stdout: 2022-04-02T03:12:45.793 INFO:teuthology.orchestra.run.smithi029.stderr:no valid command found; 10 closest matches: 2022-04-02T03:12:45.794 INFO:teuthology.orchestra.run.smithi029.stderr:pg stat 2022-04-02T03:12:45.794 INFO:teuthology.orchestra.run.smithi029.stderr:pg getmap 2022-04-02T03:12:45.794 INFO:teuthology.orchestra.run.smithi029.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-04-02T03:12:45.794 INFO:teuthology.orchestra.run.smithi029.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-04-02T03:12:45.795 INFO:teuthology.orchestra.run.smithi029.stderr:pg dump_pools_json 2022-04-02T03:12:45.795 INFO:teuthology.orchestra.run.smithi029.stderr:pg ls-by-pool [...] 2022-04-02T03:12:45.795 INFO:teuthology.orchestra.run.smithi029.stderr:pg ls-by-primary [] [...] 2022-04-02T03:12:45.795 INFO:teuthology.orchestra.run.smithi029.stderr:pg ls-by-osd [] [...] 2022-04-02T03:12:45.795 INFO:teuthology.orchestra.run.smithi029.stderr:pg ls [] [...] 2022-04-02T03:12:45.796 INFO:teuthology.orchestra.run.smithi029.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-04-02T03:12:45.796 INFO:teuthology.orchestra.run.smithi029.stderr:Error EINVAL: invalid command 2022-04-02T03:12:45.798 INFO:teuthology.orchestra.run.smithi029.stderr:command terminated with exit code 22 2022-04-02T03:12:45.800 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-04-02T03:12:47.657 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:12:47.658484 I | op-mon: mon c is not yet running 2022-04-02T03:12:47.657 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:12:47.658522 I | op-mon: mons running: [a b] 2022-04-02T03:12:54.179 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:12:54.180940 I | op-mon: mon c is not yet running 2022-04-02T03:12:54.180 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:12:54.180982 I | op-mon: mons running: [a b] 2022-04-02T03:12:55.802 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph orch status -f json 2022-04-02T03:12:56.720 INFO:teuthology.orchestra.run.smithi029.stdout: 2022-04-02T03:12:56.720 INFO:teuthology.orchestra.run.smithi029.stderr:no valid command found; 10 closest matches: 2022-04-02T03:12:56.721 INFO:teuthology.orchestra.run.smithi029.stderr:pg stat 2022-04-02T03:12:56.721 INFO:teuthology.orchestra.run.smithi029.stderr:pg getmap 2022-04-02T03:12:56.721 INFO:teuthology.orchestra.run.smithi029.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-04-02T03:12:56.721 INFO:teuthology.orchestra.run.smithi029.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-04-02T03:12:56.722 INFO:teuthology.orchestra.run.smithi029.stderr:pg dump_pools_json 2022-04-02T03:12:56.722 INFO:teuthology.orchestra.run.smithi029.stderr:pg ls-by-pool [...] 2022-04-02T03:12:56.722 INFO:teuthology.orchestra.run.smithi029.stderr:pg ls-by-primary [] [...] 2022-04-02T03:12:56.722 INFO:teuthology.orchestra.run.smithi029.stderr:pg ls-by-osd [] [...] 2022-04-02T03:12:56.722 INFO:teuthology.orchestra.run.smithi029.stderr:pg ls [] [...] 2022-04-02T03:12:56.723 INFO:teuthology.orchestra.run.smithi029.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-04-02T03:12:56.723 INFO:teuthology.orchestra.run.smithi029.stderr:Error EINVAL: invalid command 2022-04-02T03:12:56.724 INFO:teuthology.orchestra.run.smithi029.stderr:command terminated with exit code 22 2022-04-02T03:12:56.726 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-04-02T03:12:59.281 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:12:59.282664 I | op-mon: mon c is not yet running 2022-04-02T03:12:59.281 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:12:59.282702 I | op-mon: mons running: [a b] 2022-04-02T03:13:04.319 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:13:04.321414 I | op-mon: mon c is not yet running 2022-04-02T03:13:04.320 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:13:04.321453 I | op-mon: mons running: [a b] 2022-04-02T03:13:06.728 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph orch status -f json 2022-04-02T03:13:07.222 INFO:teuthology.orchestra.run.smithi029.stdout: 2022-04-02T03:13:07.223 INFO:teuthology.orchestra.run.smithi029.stderr:no valid command found; 10 closest matches: 2022-04-02T03:13:07.223 INFO:teuthology.orchestra.run.smithi029.stderr:pg stat 2022-04-02T03:13:07.223 INFO:teuthology.orchestra.run.smithi029.stderr:pg getmap 2022-04-02T03:13:07.223 INFO:teuthology.orchestra.run.smithi029.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-04-02T03:13:07.224 INFO:teuthology.orchestra.run.smithi029.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-04-02T03:13:07.224 INFO:teuthology.orchestra.run.smithi029.stderr:pg dump_pools_json 2022-04-02T03:13:07.224 INFO:teuthology.orchestra.run.smithi029.stderr:pg ls-by-pool [...] 2022-04-02T03:13:07.224 INFO:teuthology.orchestra.run.smithi029.stderr:pg ls-by-primary [] [...] 2022-04-02T03:13:07.224 INFO:teuthology.orchestra.run.smithi029.stderr:pg ls-by-osd [] [...] 2022-04-02T03:13:07.225 INFO:teuthology.orchestra.run.smithi029.stderr:pg ls [] [...] 2022-04-02T03:13:07.225 INFO:teuthology.orchestra.run.smithi029.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-04-02T03:13:07.225 INFO:teuthology.orchestra.run.smithi029.stderr:Error EINVAL: invalid command 2022-04-02T03:13:07.227 INFO:teuthology.orchestra.run.smithi029.stderr:command terminated with exit code 22 2022-04-02T03:13:07.231 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-04-02T03:13:09.348 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:13:09.349679 I | op-mon: mon c is not yet running 2022-04-02T03:13:09.348 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:13:09.349708 I | op-mon: mons running: [a b] 2022-04-02T03:13:14.368 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:13:14.370478 I | op-mon: mons running: [a b c] 2022-04-02T03:13:16.018 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:13:16.020185 I | op-mon: Monitors in quorum: [a b c] 2022-04-02T03:13:16.019 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:13:16.020227 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-04-02T03:13:16.238 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:13:16.239878 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-04-02T03:13:16.238 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:13:16.239913 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-04-02T03:13:16.534 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:13:16.536487 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-04-02T03:13:16.535 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:13:16.536518 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-04-02T03:13:16.758 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:13:16.760343 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-04-02T03:13:16.759 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:13:16.760395 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-04-02T03:13:16.987 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:13:16.989355 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-04-02T03:13:16.988 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:13:16.989383 I | op-config: deleting "log file" option from the mon configuration database 2022-04-02T03:13:17.201 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:13:17.202706 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-04-02T03:13:17.202 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:13:17.202749 I | cephclient: getting or creating ceph auth key "client.csi-rbd-provisioner" 2022-04-02T03:13:17.233 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph orch status -f json 2022-04-02T03:13:17.555 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:13:17.556618 I | cephclient: getting or creating ceph auth key "client.csi-rbd-node" 2022-04-02T03:13:17.709 INFO:teuthology.orchestra.run.smithi029.stdout: 2022-04-02T03:13:17.710 INFO:teuthology.orchestra.run.smithi029.stderr:no valid command found; 10 closest matches: 2022-04-02T03:13:17.710 INFO:teuthology.orchestra.run.smithi029.stderr:pg stat 2022-04-02T03:13:17.711 INFO:teuthology.orchestra.run.smithi029.stderr:pg getmap 2022-04-02T03:13:17.711 INFO:teuthology.orchestra.run.smithi029.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-04-02T03:13:17.711 INFO:teuthology.orchestra.run.smithi029.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-04-02T03:13:17.711 INFO:teuthology.orchestra.run.smithi029.stderr:pg dump_pools_json 2022-04-02T03:13:17.711 INFO:teuthology.orchestra.run.smithi029.stderr:pg ls-by-pool [...] 2022-04-02T03:13:17.712 INFO:teuthology.orchestra.run.smithi029.stderr:pg ls-by-primary [] [...] 2022-04-02T03:13:17.712 INFO:teuthology.orchestra.run.smithi029.stderr:pg ls-by-osd [] [...] 2022-04-02T03:13:17.712 INFO:teuthology.orchestra.run.smithi029.stderr:pg ls [] [...] 2022-04-02T03:13:17.712 INFO:teuthology.orchestra.run.smithi029.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-04-02T03:13:17.713 INFO:teuthology.orchestra.run.smithi029.stderr:Error EINVAL: invalid command 2022-04-02T03:13:17.713 INFO:teuthology.orchestra.run.smithi029.stderr:command terminated with exit code 22 2022-04-02T03:13:17.715 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-04-02T03:13:17.923 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:13:17.925626 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-provisioner" 2022-04-02T03:13:18.283 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:13:18.284780 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-node" 2022-04-02T03:13:18.787 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:13:18.788956 I | ceph-csi: created kubernetes csi secrets for cluster "rook-ceph" 2022-04-02T03:13:18.787 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:13:18.788990 I | cephclient: getting or creating ceph auth key "client.crash" 2022-04-02T03:13:19.228 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:13:19.230543 I | ceph-crashcollector-controller: created kubernetes crash collector secret for cluster "rook-ceph" 2022-04-02T03:13:19.229 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:13:19.230581 I | cephclient: create rbd-mirror bootstrap peer token "client.rbd-mirror-peer" 2022-04-02T03:13:19.229 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:13:19.230591 I | cephclient: getting or creating ceph auth key "client.rbd-mirror-peer" 2022-04-02T03:13:19.592 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:13:19.594526 I | cephclient: successfully created rbd-mirror bootstrap peer token for cluster "rook-ceph" 2022-04-02T03:13:19.611 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:13:19.613180 I | op-mgr: start running mgr 2022-04-02T03:13:19.611 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:13:19.613219 I | cephclient: getting or creating ceph auth key "mgr.a" 2022-04-02T03:13:22.087 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:13:22.088876 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi142": the object has been modified; please apply your changes to the latest version and try again 2022-04-02T03:13:22.365 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:13:22.367064 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi142": the object has been modified; please apply your changes to the latest version and try again 2022-04-02T03:13:27.060 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:13:27.061752 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi104": the object has been modified; please apply your changes to the latest version and try again 2022-04-02T03:13:27.669 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:13:27.671518 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi104": the object has been modified; please apply your changes to the latest version and try again 2022-04-02T03:13:27.717 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph orch status -f json 2022-04-02T03:13:28.826 INFO:teuthology.orchestra.run.smithi029.stdout: 2022-04-02T03:13:28.827 INFO:teuthology.orchestra.run.smithi029.stderr:no valid command found; 10 closest matches: 2022-04-02T03:13:28.827 INFO:teuthology.orchestra.run.smithi029.stderr:pg stat 2022-04-02T03:13:28.827 INFO:teuthology.orchestra.run.smithi029.stderr:pg getmap 2022-04-02T03:13:28.828 INFO:teuthology.orchestra.run.smithi029.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-04-02T03:13:28.828 INFO:teuthology.orchestra.run.smithi029.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-04-02T03:13:28.828 INFO:teuthology.orchestra.run.smithi029.stderr:pg dump_pools_json 2022-04-02T03:13:28.828 INFO:teuthology.orchestra.run.smithi029.stderr:pg ls-by-pool [...] 2022-04-02T03:13:28.829 INFO:teuthology.orchestra.run.smithi029.stderr:pg ls-by-primary [] [...] 2022-04-02T03:13:28.829 INFO:teuthology.orchestra.run.smithi029.stderr:pg ls-by-osd [] [...] 2022-04-02T03:13:28.829 INFO:teuthology.orchestra.run.smithi029.stderr:pg ls [] [...] 2022-04-02T03:13:28.829 INFO:teuthology.orchestra.run.smithi029.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-04-02T03:13:28.829 INFO:teuthology.orchestra.run.smithi029.stderr:Error EINVAL: invalid command 2022-04-02T03:13:28.830 INFO:teuthology.orchestra.run.smithi029.stderr:command terminated with exit code 22 2022-04-02T03:13:28.831 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-04-02T03:13:38.833 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph orch status -f json 2022-04-02T03:13:39.607 INFO:teuthology.orchestra.run.smithi029.stdout: 2022-04-02T03:13:39.608 INFO:teuthology.orchestra.run.smithi029.stderr:Error ENOENT: No orchestrator configured (try `ceph orch set backend`) 2022-04-02T03:13:39.613 INFO:teuthology.orchestra.run.smithi029.stderr:command terminated with exit code 2 2022-04-02T03:13:39.616 DEBUG:teuthology.orchestra.run:got remote process result: 2 2022-04-02T03:13:41.680 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:13:41.682602 I | op-k8sutil: finished waiting for updated deployment "rook-ceph-mgr-a" 2022-04-02T03:13:41.684 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:13:41.686983 I | op-mgr: setting services to point to mgr "a" 2022-04-02T03:13:41.727 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:13:41.729672 I | op-mgr: no need to update service "rook-ceph-mgr" 2022-04-02T03:13:41.727 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:13:41.729747 I | op-mgr: successful modules: balancer 2022-04-02T03:13:41.735 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:13:41.737369 I | op-osd: start running osds in namespace "rook-ceph" 2022-04-02T03:13:41.735 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:13:41.737415 W | op-osd: useAllNodes is set to false and no nodes, storageClassDevicesets or volumeSources are specified, no OSD pods are going to be created 2022-04-02T03:13:41.735 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:13:41.737426 I | op-osd: wait timeout for healthy OSDs during upgrade or restart is "10m0s" 2022-04-02T03:13:41.793 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:13:41.795963 I | op-osd: start provisioning the OSDs on PVCs, if needed 2022-04-02T03:13:41.801 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:13:41.801311 I | op-osd: no storageClassDeviceSets defined to configure OSDs on PVCs 2022-04-02T03:13:41.801 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:13:41.801589 I | op-osd: start provisioning the OSDs on nodes, if needed 2022-04-02T03:13:41.802 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:13:41.801605 I | op-osd: no nodes are defined for configuring OSDs on raw devices 2022-04-02T03:13:42.204 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:13:42.206147 I | op-osd: finished running OSDs in namespace "rook-ceph" 2022-04-02T03:13:42.204 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:13:42.206179 I | ceph-cluster-controller: done reconciling ceph cluster in namespace "rook-ceph" 2022-04-02T03:13:42.216 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:13:42.218885 I | ceph-cluster-controller: enabling ceph mon monitoring goroutine for cluster "rook-ceph" 2022-04-02T03:13:42.217 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:13:42.218925 I | ceph-cluster-controller: enabling ceph osd monitoring goroutine for cluster "rook-ceph" 2022-04-02T03:13:42.217 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:13:42.218947 I | ceph-cluster-controller: enabling ceph status monitoring goroutine for cluster "rook-ceph" 2022-04-02T03:13:43.249 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:13:43.251821 I | op-mgr: successful modules: mgr module(s) from the spec 2022-04-02T03:13:43.252 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:13:43.252806 I | op-mgr: successful modules: dashboard 2022-04-02T03:13:43.253 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:13:43.253415 I | op-mgr: successful modules: prometheus 2022-04-02T03:13:49.617 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph orch status -f json 2022-04-02T03:13:51.984 INFO:teuthology.orchestra.run.smithi029.stdout: 2022-04-02T03:13:51.987 INFO:teuthology.orchestra.run.smithi029.stderr:Error ENOENT: No orchestrator configured (try `ceph orch set backend`) 2022-04-02T03:13:51.991 INFO:teuthology.orchestra.run.smithi029.stderr:command terminated with exit code 2 2022-04-02T03:13:51.993 DEBUG:teuthology.orchestra.run:got remote process result: 2 2022-04-02T03:13:52.131 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:13:52.133340 I | op-mgr: successful modules: orchestrator modules 2022-04-02T03:14:01.995 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph orch status -f json 2022-04-02T03:14:02.638 INFO:teuthology.orchestra.run.smithi029.stdout: 2022-04-02T03:14:02.638 INFO:teuthology.orchestra.run.smithi029.stdout:{"available": true, "backend": "rook"} 2022-04-02T03:14:02.660 INFO:tasks.rook: mgr/rook orchestrator is active 2022-04-02T03:14:02.661 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph config set mgr mgr/rook/storage_class scratch 2022-04-02T03:14:03.309 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph orch apply osd --all-available-devices 2022-04-02T03:14:04.088 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:04.089871 I | ceph-cluster-controller: CR has changed for "rook-ceph". diff= v1.ClusterSpec{ 2022-04-02T03:14:04.088 INFO:tasks.rook.operator.smithi029.stdout: CephVersion: {Image: "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", AllowUnsupported: true}, 2022-04-02T03:14:04.089 INFO:tasks.rook.operator.smithi029.stdout: Storage: v1.StorageScopeSpec{ 2022-04-02T03:14:04.089 INFO:tasks.rook.operator.smithi029.stdout: ... // 3 identical fields 2022-04-02T03:14:04.089 INFO:tasks.rook.operator.smithi029.stdout: Config: nil, 2022-04-02T03:14:04.089 INFO:tasks.rook.operator.smithi029.stdout: Selection: {}, 2022-04-02T03:14:04.089 INFO:tasks.rook.operator.smithi029.stdout:- StorageClassDeviceSets: nil, 2022-04-02T03:14:04.090 INFO:tasks.rook.operator.smithi029.stdout:+ StorageClassDeviceSets: []v1.StorageClassDeviceSet{ 2022-04-02T03:14:04.090 INFO:tasks.rook.operator.smithi029.stdout:+ { 2022-04-02T03:14:04.090 INFO:tasks.rook.operator.smithi029.stdout:+ Name: "smithi029-lv-1", 2022-04-02T03:14:04.090 INFO:tasks.rook.operator.smithi029.stdout:+ Count: 1, 2022-04-02T03:14:04.090 INFO:tasks.rook.operator.smithi029.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-04-02T03:14:04.091 INFO:tasks.rook.operator.smithi029.stdout:+ }, 2022-04-02T03:14:04.091 INFO:tasks.rook.operator.smithi029.stdout:+ { 2022-04-02T03:14:04.092 INFO:tasks.rook.operator.smithi029.stdout:+ Name: "smithi029-lv-2", 2022-04-02T03:14:04.092 INFO:tasks.rook.operator.smithi029.stdout:+ Count: 1, 2022-04-02T03:14:04.092 INFO:tasks.rook.operator.smithi029.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-04-02T03:14:04.093 INFO:tasks.rook.operator.smithi029.stdout:+ }, 2022-04-02T03:14:04.093 INFO:tasks.rook.operator.smithi029.stdout:+ { 2022-04-02T03:14:04.093 INFO:tasks.rook.operator.smithi029.stdout:+ Name: "smithi029-lv-3", 2022-04-02T03:14:04.093 INFO:tasks.rook.operator.smithi029.stdout:+ Count: 1, 2022-04-02T03:14:04.093 INFO:tasks.rook.operator.smithi029.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-04-02T03:14:04.094 INFO:tasks.rook.operator.smithi029.stdout:+ }, 2022-04-02T03:14:04.094 INFO:teuthology.orchestra.run.smithi029.stdout:['Success'] 2022-04-02T03:14:04.099 INFO:tasks.rook.operator.smithi029.stdout:+ { 2022-04-02T03:14:04.099 INFO:tasks.rook.operator.smithi029.stdout:+ Name: "smithi029-lv-4", 2022-04-02T03:14:04.100 INFO:tasks.rook.operator.smithi029.stdout:+ Count: 1, 2022-04-02T03:14:04.100 INFO:tasks.rook.operator.smithi029.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-04-02T03:14:04.100 INFO:tasks.rook.operator.smithi029.stdout:+ }, 2022-04-02T03:14:04.100 INFO:tasks.rook.operator.smithi029.stdout:+ { 2022-04-02T03:14:04.101 INFO:tasks.rook.operator.smithi029.stdout:+ Name: "smithi104-lv-1", 2022-04-02T03:14:04.101 INFO:tasks.rook.operator.smithi029.stdout:+ Count: 1, 2022-04-02T03:14:04.101 INFO:tasks.rook.operator.smithi029.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-04-02T03:14:04.101 INFO:tasks.rook.operator.smithi029.stdout:+ }, 2022-04-02T03:14:04.101 INFO:tasks.rook.operator.smithi029.stdout:+ { 2022-04-02T03:14:04.102 INFO:tasks.rook.operator.smithi029.stdout:+ Name: "smithi104-lv-2", 2022-04-02T03:14:04.102 INFO:tasks.rook.operator.smithi029.stdout:+ Count: 1, 2022-04-02T03:14:04.102 INFO:tasks.rook.operator.smithi029.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-04-02T03:14:04.102 INFO:tasks.rook.operator.smithi029.stdout:+ }, 2022-04-02T03:14:04.102 INFO:tasks.rook.operator.smithi029.stdout:+ { 2022-04-02T03:14:04.103 INFO:tasks.rook.operator.smithi029.stdout:+ Name: "smithi104-lv-3", 2022-04-02T03:14:04.103 INFO:tasks.rook.operator.smithi029.stdout:+ Count: 1, 2022-04-02T03:14:04.103 INFO:tasks.rook.operator.smithi029.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-04-02T03:14:04.103 INFO:tasks.rook.operator.smithi029.stdout:+ }, 2022-04-02T03:14:04.104 INFO:tasks.rook.operator.smithi029.stdout:+ { 2022-04-02T03:14:04.104 INFO:tasks.rook.operator.smithi029.stdout:+ Name: "smithi104-lv-4", 2022-04-02T03:14:04.104 INFO:tasks.rook.operator.smithi029.stdout:+ Count: 1, 2022-04-02T03:14:04.104 INFO:tasks.rook.operator.smithi029.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-04-02T03:14:04.104 INFO:tasks.rook.operator.smithi029.stdout:+ }, 2022-04-02T03:14:04.105 INFO:tasks.rook.operator.smithi029.stdout:+ { 2022-04-02T03:14:04.105 INFO:tasks.rook.operator.smithi029.stdout:+ Name: "smithi142-lv-1", 2022-04-02T03:14:04.105 INFO:tasks.rook.operator.smithi029.stdout:+ Count: 1, 2022-04-02T03:14:04.105 INFO:tasks.rook.operator.smithi029.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-04-02T03:14:04.105 INFO:tasks.rook.operator.smithi029.stdout:+ }, 2022-04-02T03:14:04.106 INFO:tasks.rook.operator.smithi029.stdout:+ { 2022-04-02T03:14:04.106 INFO:tasks.rook.operator.smithi029.stdout:+ Name: "smithi142-lv-2", 2022-04-02T03:14:04.106 INFO:tasks.rook.operator.smithi029.stdout:+ Count: 1, 2022-04-02T03:14:04.106 INFO:tasks.rook.operator.smithi029.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-04-02T03:14:04.107 INFO:tasks.rook.operator.smithi029.stdout:+ }, 2022-04-02T03:14:04.107 INFO:tasks.rook.operator.smithi029.stdout:+ { 2022-04-02T03:14:04.107 INFO:tasks.rook.operator.smithi029.stdout:+ Name: "smithi142-lv-3", 2022-04-02T03:14:04.107 INFO:tasks.rook.operator.smithi029.stdout:+ Count: 1, 2022-04-02T03:14:04.107 INFO:tasks.rook.operator.smithi029.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-04-02T03:14:04.108 INFO:tasks.rook.operator.smithi029.stdout:+ }, 2022-04-02T03:14:04.108 INFO:tasks.rook.operator.smithi029.stdout:+ { 2022-04-02T03:14:04.108 INFO:tasks.rook.operator.smithi029.stdout:+ Name: "smithi142-lv-4", 2022-04-02T03:14:04.108 INFO:tasks.rook.operator.smithi029.stdout:+ Count: 1, 2022-04-02T03:14:04.108 INFO:tasks.rook.operator.smithi029.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-04-02T03:14:04.109 INFO:tasks.rook.operator.smithi029.stdout:+ }, 2022-04-02T03:14:04.109 INFO:tasks.rook.operator.smithi029.stdout:+ }, 2022-04-02T03:14:04.109 INFO:tasks.rook.operator.smithi029.stdout: }, 2022-04-02T03:14:04.109 INFO:tasks.rook.operator.smithi029.stdout: Annotations: nil, 2022-04-02T03:14:04.110 INFO:tasks.rook.operator.smithi029.stdout: Labels: nil, 2022-04-02T03:14:04.110 INFO:tasks.rook.operator.smithi029.stdout: ... // 20 identical fields 2022-04-02T03:14:04.110 INFO:tasks.rook.operator.smithi029.stdout: } 2022-04-02T03:14:04.110 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:04.090033 I | operator: reloading operator's CRDs manager, cancelling all orchestrations! 2022-04-02T03:14:04.110 INFO:tasks.rook.operator.smithi029.stdout:I0402 03:14:04.090432 1 manager.go:148] objectbucket.io/provisioner-manager "msg"="stopping provisioner" "name"="rook-ceph.ceph.rook.io/bucket" "reason"={} 2022-04-02T03:14:04.111 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:04.090486 I | ceph-cluster-controller: stopping monitoring of ceph status 2022-04-02T03:14:04.111 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:04.090519 I | op-mon: stopping monitoring of mons in namespace "rook-ceph" 2022-04-02T03:14:04.111 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:04.090531 I | op-osd: stopping monitoring of OSDs in namespace "rook-ceph" 2022-04-02T03:14:04.111 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:04.090572 I | operator: successfully started the controller-runtime manager 2022-04-02T03:14:04.117 INFO:tasks.rook:Waiting for 12 OSDs 2022-04-02T03:14:04.117 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph osd stat -f json 2022-04-02T03:14:04.147 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:04.150634 I | op-k8sutil: ROOK_CURRENT_NAMESPACE_ONLY="false" (env var) 2022-04-02T03:14:04.148 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:04.150666 I | operator: watching all namespaces for Ceph CRs 2022-04-02T03:14:04.148 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:04.150726 I | operator: setting up schemes 2022-04-02T03:14:04.151 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:04.154948 I | operator: setting up the controller-runtime manager 2022-04-02T03:14:04.355 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:04.358147 I | operator: looking for admission webhook secret "rook-ceph-admission-controller" 2022-04-02T03:14:04.360 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:04.360472 I | operator: admission webhook secret "rook-ceph-admission-controller" not found. proceeding without the admission controller 2022-04-02T03:14:04.360 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:04.360498 I | ceph-cluster-controller: successfully started 2022-04-02T03:14:04.362 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:04.360532 I | ceph-cluster-controller: enabling hotplug orchestration 2022-04-02T03:14:04.362 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:04.360549 I | ceph-crashcollector-controller: successfully started 2022-04-02T03:14:04.363 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:04.360569 I | ceph-block-pool-controller: successfully started 2022-04-02T03:14:04.363 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:04.360583 I | ceph-object-store-user-controller: successfully started 2022-04-02T03:14:04.363 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:04.360600 I | ceph-object-realm-controller: successfully started 2022-04-02T03:14:04.363 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:04.360610 I | ceph-object-zonegroup-controller: successfully started 2022-04-02T03:14:04.364 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:04.360619 I | ceph-object-zone-controller: successfully started 2022-04-02T03:14:04.364 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:04.360700 I | ceph-object-controller: successfully started 2022-04-02T03:14:04.364 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:04.360721 I | ceph-file-controller: successfully started 2022-04-02T03:14:04.442 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:04.360745 I | ceph-nfs-controller: successfully started 2022-04-02T03:14:04.443 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:04.360768 I | ceph-rbd-mirror-controller: successfully started 2022-04-02T03:14:04.443 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:04.360794 I | ceph-client-controller: successfully started 2022-04-02T03:14:04.444 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:04.360817 I | ceph-filesystem-mirror-controller: successfully started 2022-04-02T03:14:04.444 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:04.360830 I | operator: rook-ceph-operator-config-controller successfully started 2022-04-02T03:14:04.444 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:04.360838 I | ceph-csi: rook-ceph-operator-csi-controller successfully started 2022-04-02T03:14:04.445 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:04.360845 I | op-bucket-prov: rook-ceph-operator-bucket-controller successfully started 2022-04-02T03:14:04.445 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:04.360853 I | ceph-bucket-topic: successfully started 2022-04-02T03:14:04.446 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:04.360858 I | ceph-bucket-notification: successfully started 2022-04-02T03:14:04.446 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:04.360864 I | ceph-bucket-notification: successfully started 2022-04-02T03:14:04.446 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:04.360870 I | ceph-fs-subvolumegroup-controller: successfully started 2022-04-02T03:14:04.447 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:04.361960 I | operator: starting the controller-runtime manager 2022-04-02T03:14:04.465 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:04.467405 I | op-k8sutil: ROOK_CEPH_COMMANDS_TIMEOUT_SECONDS="15" (configmap) 2022-04-02T03:14:04.465 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:04.467452 I | op-k8sutil: ROOK_LOG_LEVEL="INFO" (configmap) 2022-04-02T03:14:04.465 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:04.467462 I | op-k8sutil: ROOK_ENABLE_DISCOVERY_DAEMON="false" (configmap) 2022-04-02T03:14:04.465 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:04.467537 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-04-02T03:14:04.469 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:04.470753 I | operator: rook-ceph-operator-config-controller done reconciling 2022-04-02T03:14:04.472 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:04.472858 I | op-mon: parsing mon endpoints: a=10.102.133.199:6789,b=10.107.147.59:6789,c=10.96.208.36:6789 2022-04-02T03:14:04.473 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:04.472930 I | op-mon: parsing mon endpoints: a=10.102.133.199:6789,b=10.107.147.59:6789,c=10.96.208.36:6789 2022-04-02T03:14:04.473 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:04.472984 I | op-k8sutil: ROOK_OBC_WATCH_OPERATOR_NAMESPACE="true" (configmap) 2022-04-02T03:14:04.473 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:04.473000 I | op-bucket-prov: ceph bucket provisioner launched watching for provisioner "rook-ceph.ceph.rook.io/bucket" 2022-04-02T03:14:04.473 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:04.474607 I | op-bucket-prov: successfully reconciled bucket provisioner 2022-04-02T03:14:04.473 INFO:tasks.rook.operator.smithi029.stdout:I0402 03:14:04.474658 1 manager.go:135] objectbucket.io/provisioner-manager "msg"="starting provisioner" "name"="rook-ceph.ceph.rook.io/bucket" 2022-04-02T03:14:04.475 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:04.478168 I | ceph-cluster-controller: enabling ceph mon monitoring goroutine for cluster "rook-ceph" 2022-04-02T03:14:04.475 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:04.478194 I | ceph-cluster-controller: enabling ceph osd monitoring goroutine for cluster "rook-ceph" 2022-04-02T03:14:04.476 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:04.478204 I | ceph-cluster-controller: enabling ceph status monitoring goroutine for cluster "rook-ceph" 2022-04-02T03:14:04.480 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:04.484003 I | ceph-csi: successfully created csi config map "rook-ceph-csi-config" 2022-04-02T03:14:04.483 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:04.487092 I | ceph-spec: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a... 2022-04-02T03:14:04.543 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:04.544698 I | op-k8sutil: ROOK_CSI_ENABLE_RBD="true" (configmap) 2022-04-02T03:14:04.543 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:04.544726 I | op-k8sutil: ROOK_CSI_ENABLE_CEPHFS="true" (configmap) 2022-04-02T03:14:04.544 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:04.544733 I | op-k8sutil: ROOK_CSI_ALLOW_UNSUPPORTED_VERSION="false" (configmap) 2022-04-02T03:14:04.544 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:04.544740 I | op-k8sutil: ROOK_CSI_ENABLE_GRPC_METRICS="false" (configmap) 2022-04-02T03:14:04.544 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:04.544746 I | op-k8sutil: CSI_ENABLE_HOST_NETWORK="true" (default) 2022-04-02T03:14:04.544 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:04.544754 I | op-k8sutil: ROOK_CSI_CEPH_IMAGE="quay.io/cephcsi/cephcsi:v3.5.1" (default) 2022-04-02T03:14:04.545 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:04.544761 I | op-k8sutil: ROOK_CSI_REGISTRAR_IMAGE="k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.5.0" (default) 2022-04-02T03:14:04.545 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:04.544771 I | op-k8sutil: ROOK_CSI_PROVISIONER_IMAGE="k8s.gcr.io/sig-storage/csi-provisioner:v3.1.0" (default) 2022-04-02T03:14:04.545 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:04.544783 I | op-k8sutil: ROOK_CSI_ATTACHER_IMAGE="k8s.gcr.io/sig-storage/csi-attacher:v3.4.0" (default) 2022-04-02T03:14:04.545 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:04.544793 I | op-k8sutil: ROOK_CSI_SNAPSHOTTER_IMAGE="k8s.gcr.io/sig-storage/csi-snapshotter:v5.0.1" (default) 2022-04-02T03:14:04.545 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:04.544804 I | op-k8sutil: ROOK_CSI_KUBELET_DIR_PATH="/var/lib/kubelet" (default) 2022-04-02T03:14:04.546 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:04.544818 I | op-k8sutil: CSI_VOLUME_REPLICATION_IMAGE="quay.io/csiaddons/volumereplication-operator:v0.3.0" (default) 2022-04-02T03:14:04.546 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:04.544829 I | op-k8sutil: ROOK_CSIADDONS_IMAGE="quay.io/csiaddons/k8s-sidecar:v0.2.1" (default) 2022-04-02T03:14:04.546 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:04.544843 I | op-k8sutil: ROOK_CSI_CEPHFS_POD_LABELS="" (default) 2022-04-02T03:14:04.546 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:04.544853 I | op-k8sutil: ROOK_CSI_RBD_POD_LABELS="" (default) 2022-04-02T03:14:04.546 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:04.544866 I | ceph-csi: detecting the ceph csi image version for image "quay.io/cephcsi/cephcsi:v3.5.1" 2022-04-02T03:14:04.547 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:04.544964 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-04-02T03:14:04.547 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:04.544986 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default) 2022-04-02T03:14:04.619 INFO:teuthology.orchestra.run.smithi029.stdout: 2022-04-02T03:14:04.641 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":2,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2022-04-02T03:14:04.641 INFO:tasks.rook: have 0/12 OSDs 2022-04-02T03:14:14.642 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph osd stat -f json 2022-04-02T03:14:16.583 INFO:teuthology.orchestra.run.smithi029.stdout: 2022-04-02T03:14:16.605 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":2,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2022-04-02T03:14:16.605 INFO:tasks.rook: have 0/12 OSDs 2022-04-02T03:14:18.629 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:18.630249 I | ceph-csi: Detected ceph CSI image version: "v3.5.1" 2022-04-02T03:14:18.630 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:18.630288 I | op-k8sutil: CSI_FORCE_CEPHFS_KERNEL_CLIENT="true" (configmap) 2022-04-02T03:14:18.630 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:18.630301 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default) 2022-04-02T03:14:18.630 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:18.630308 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default) 2022-04-02T03:14:18.630 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:18.630316 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default) 2022-04-02T03:14:18.631 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:18.630322 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default) 2022-04-02T03:14:18.631 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:18.630329 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default) 2022-04-02T03:14:18.631 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:18.630335 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default) 2022-04-02T03:14:18.631 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:18.630343 I | op-k8sutil: CSIADDONS_PORT="9070" (default) 2022-04-02T03:14:18.632 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:18.630350 I | op-k8sutil: CSIADDONS_PORT="9070" (default) 2022-04-02T03:14:18.632 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:18.630357 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default) 2022-04-02T03:14:18.632 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:18.630364 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default) 2022-04-02T03:14:18.632 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:18.630370 I | op-k8sutil: CSI_PLUGIN_PRIORITY_CLASSNAME="" (default) 2022-04-02T03:14:18.633 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:18.630377 I | op-k8sutil: CSI_PROVISIONER_PRIORITY_CLASSNAME="" (default) 2022-04-02T03:14:18.633 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:18.630384 I | op-k8sutil: CSI_ENABLE_OMAP_GENERATOR="false" (default) 2022-04-02T03:14:18.633 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:18.630392 I | op-k8sutil: CSI_ENABLE_RBD_SNAPSHOTTER="true" (configmap) 2022-04-02T03:14:18.633 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:18.630398 I | op-k8sutil: CSI_ENABLE_CEPHFS_SNAPSHOTTER="true" (configmap) 2022-04-02T03:14:18.634 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:18.630405 I | op-k8sutil: CSI_ENABLE_VOLUME_REPLICATION="false" (configmap) 2022-04-02T03:14:18.634 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:18.630414 I | op-k8sutil: CSI_ENABLE_CSIADDONS="false" (configmap) 2022-04-02T03:14:18.634 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:18.630427 I | op-k8sutil: CSI_CEPHFS_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-04-02T03:14:18.634 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:18.630441 I | op-k8sutil: CSI_RBD_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-04-02T03:14:18.634 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:18.630452 I | op-k8sutil: CSI_PLUGIN_ENABLE_SELINUX_HOST_MOUNT="false" (configmap) 2022-04-02T03:14:18.635 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:18.630463 I | ceph-csi: Kubernetes version is 1.23 2022-04-02T03:14:18.635 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:18.630475 I | op-k8sutil: ROOK_CSI_RESIZER_IMAGE="k8s.gcr.io/sig-storage/csi-resizer:v1.4.0" (default) 2022-04-02T03:14:18.635 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:18.630488 I | op-k8sutil: CSI_LOG_LEVEL="" (default) 2022-04-02T03:14:18.657 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:18.660566 I | op-k8sutil: CSI_PROVISIONER_REPLICAS="2" (configmap) 2022-04-02T03:14:18.667 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:18.670576 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-04-02T03:14:18.668 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:18.670603 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default) 2022-04-02T03:14:18.668 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:18.670609 I | op-k8sutil: CSI_PLUGIN_TOLERATIONS="" (default) 2022-04-02T03:14:18.668 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:18.670613 I | op-k8sutil: CSI_PLUGIN_NODE_AFFINITY="" (default) 2022-04-02T03:14:18.668 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:18.670617 I | op-k8sutil: CSI_RBD_PLUGIN_TOLERATIONS="" (default) 2022-04-02T03:14:18.669 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:18.670621 I | op-k8sutil: CSI_RBD_PLUGIN_NODE_AFFINITY="" (default) 2022-04-02T03:14:18.669 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:18.670626 I | op-k8sutil: CSI_RBD_PLUGIN_RESOURCE="" (default) 2022-04-02T03:14:19.072 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:19.075415 I | op-k8sutil: CSI_RBD_PROVISIONER_TOLERATIONS="" (default) 2022-04-02T03:14:19.072 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:19.075448 I | op-k8sutil: CSI_RBD_PROVISIONER_NODE_AFFINITY="" (default) 2022-04-02T03:14:19.073 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:19.075457 I | op-k8sutil: CSI_RBD_PROVISIONER_RESOURCE="" (default) 2022-04-02T03:14:19.252 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:19.255429 I | ceph-csi: successfully started CSI Ceph RBD driver 2022-04-02T03:14:20.230 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:20.233491 I | op-k8sutil: CSI_CEPHFS_PLUGIN_TOLERATIONS="" (default) 2022-04-02T03:14:20.230 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:20.233524 I | op-k8sutil: CSI_CEPHFS_PLUGIN_NODE_AFFINITY="" (default) 2022-04-02T03:14:20.231 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:20.233534 I | op-k8sutil: CSI_CEPHFS_PLUGIN_RESOURCE="" (default) 2022-04-02T03:14:20.301 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:20.303797 I | ceph-spec: detected ceph image version: "17.0.0-11449 quincy" 2022-04-02T03:14:20.301 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:20.303822 I | ceph-cluster-controller: validating ceph version from provided image 2022-04-02T03:14:20.604 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:20.607567 I | op-mon: parsing mon endpoints: a=10.102.133.199:6789,b=10.107.147.59:6789,c=10.96.208.36:6789 2022-04-02T03:14:20.610 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:20.608799 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_TOLERATIONS="" (default) 2022-04-02T03:14:20.610 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:20.608827 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_NODE_AFFINITY="" (default) 2022-04-02T03:14:20.610 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:20.608835 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_RESOURCE="" (default) 2022-04-02T03:14:20.611 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:20.610921 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-04-02T03:14:20.611 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:20.611205 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-04-02T03:14:20.678 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:20.681130 I | ceph-csi: successfully started CSI CephFS driver 2022-04-02T03:14:20.938 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:20.941255 I | op-k8sutil: CSI_RBD_FSGROUPPOLICY="ReadWriteOnceWithFSType" (configmap) 2022-04-02T03:14:20.971 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:20.973493 I | ceph-csi: CSIDriver object updated for driver "rook-ceph.rbd.csi.ceph.com" 2022-04-02T03:14:20.971 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:20.973513 I | op-k8sutil: CSI_CEPHFS_FSGROUPPOLICY="ReadWriteOnceWithFSType" (configmap) 2022-04-02T03:14:20.976 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:20.979503 I | ceph-csi: CSIDriver object updated for driver "rook-ceph.cephfs.csi.ceph.com" 2022-04-02T03:14:21.058 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:21.061367 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-11449 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a" 2022-04-02T03:14:21.727 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:21.729939 I | op-mon: start running mons 2022-04-02T03:14:21.890 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:21.893555 I | op-mon: parsing mon endpoints: a=10.102.133.199:6789,b=10.107.147.59:6789,c=10.96.208.36:6789 2022-04-02T03:14:21.915 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:21.918584 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.102.133.199:6789","10.107.147.59:6789","10.96.208.36:6789"]}] data:a=10.102.133.199:6789,b=10.107.147.59:6789,c=10.96.208.36:6789 mapping:{"node":{"a":{"Name":"smithi104","Hostname":"smithi104","Address":"172.21.15.104"},"b":{"Name":"smithi142","Hostname":"smithi142","Address":"172.21.15.142"},"c":{"Name":"smithi029","Hostname":"smithi029","Address":"172.21.15.29"}}} maxMonId:2] 2022-04-02T03:14:22.030 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:22.033044 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-04-02T03:14:22.034 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:22.033363 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-04-02T03:14:23.632 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:23.635713 I | op-mon: targeting the mon count 3 2022-04-02T03:14:23.637 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:23.641290 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-04-02T03:14:24.027 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:24.030118 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-04-02T03:14:24.027 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:24.030160 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-04-02T03:14:24.407 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:24.410569 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-04-02T03:14:24.408 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:24.410600 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-04-02T03:14:24.755 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:24.758839 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-04-02T03:14:24.756 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:24.758869 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-04-02T03:14:25.104 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:25.107085 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-04-02T03:14:25.104 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:25.107116 I | op-config: deleting "log file" option from the mon configuration database 2022-04-02T03:14:25.482 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:25.485552 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-04-02T03:14:25.483 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:25.485607 I | op-mon: checking for basic quorum with existing mons 2022-04-02T03:14:25.649 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:25.652255 I | op-mon: mon "b" ip is 10.107.147.59 2022-04-02T03:14:25.737 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:25.740474 I | op-mon: mon "c" ip is 10.96.208.36 2022-04-02T03:14:26.032 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:26.035044 I | op-mon: mon "a" ip is 10.102.133.199 2022-04-02T03:14:26.606 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph osd stat -f json 2022-04-02T03:14:26.632 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:26.635514 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.102.133.199:6789","10.107.147.59:6789","10.96.208.36:6789"]}] data:a=10.102.133.199:6789,b=10.107.147.59:6789,c=10.96.208.36:6789 mapping:{"node":{"a":{"Name":"smithi104","Hostname":"smithi104","Address":"172.21.15.104"},"b":{"Name":"smithi142","Hostname":"smithi142","Address":"172.21.15.142"},"c":{"Name":"smithi029","Hostname":"smithi029","Address":"172.21.15.29"}}} maxMonId:2] 2022-04-02T03:14:27.114 INFO:teuthology.orchestra.run.smithi029.stdout: 2022-04-02T03:14:27.133 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":2,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2022-04-02T03:14:27.133 INFO:tasks.rook: have 0/12 OSDs 2022-04-02T03:14:27.231 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:27.234666 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-04-02T03:14:27.232 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:27.234973 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-04-02T03:14:27.693 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:27.696933 I | op-mon: deployment for mon rook-ceph-mon-b already exists. updating if needed 2022-04-02T03:14:27.707 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:27.709094 I | op-k8sutil: deployment "rook-ceph-mon-b" did not change, nothing to update 2022-04-02T03:14:27.707 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:27.709163 I | op-mon: waiting for mon quorum with [b c a] 2022-04-02T03:14:28.239 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:28.241992 I | op-mon: mons running: [b c a] 2022-04-02T03:14:28.667 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:28.670279 I | op-mon: Monitors in quorum: [a b c] 2022-04-02T03:14:28.676 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:28.679503 I | op-mon: deployment for mon rook-ceph-mon-c already exists. updating if needed 2022-04-02T03:14:28.687 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:28.690244 I | op-k8sutil: deployment "rook-ceph-mon-c" did not change, nothing to update 2022-04-02T03:14:28.687 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:28.690274 I | op-mon: waiting for mon quorum with [b c a] 2022-04-02T03:14:28.833 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:28.836730 I | op-mon: mons running: [b c a] 2022-04-02T03:14:29.247 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:29.250001 I | op-mon: Monitors in quorum: [a b c] 2022-04-02T03:14:29.255 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:29.258847 I | op-mon: deployment for mon rook-ceph-mon-a already exists. updating if needed 2022-04-02T03:14:29.264 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:29.267356 I | op-k8sutil: deployment "rook-ceph-mon-a" did not change, nothing to update 2022-04-02T03:14:29.264 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:29.267376 I | op-mon: waiting for mon quorum with [b c a] 2022-04-02T03:14:29.435 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:29.438890 I | op-mon: mons running: [b c a] 2022-04-02T03:14:29.843 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:29.846563 I | op-mon: Monitors in quorum: [a b c] 2022-04-02T03:14:29.843 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:29.846596 I | op-mon: mons created: 3 2022-04-02T03:14:30.268 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:30.271235 I | op-mon: waiting for mon quorum with [b c a] 2022-04-02T03:14:30.340 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:30.344021 I | op-mon: mons running: [b c a] 2022-04-02T03:14:30.784 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:30.786856 I | op-mon: Monitors in quorum: [a b c] 2022-04-02T03:14:30.784 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:30.786904 I | cephclient: getting or creating ceph auth key "client.csi-rbd-provisioner" 2022-04-02T03:14:31.283 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:31.286763 I | cephclient: getting or creating ceph auth key "client.csi-rbd-node" 2022-04-02T03:14:31.708 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:31.711555 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-provisioner" 2022-04-02T03:14:32.129 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:32.132293 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-node" 2022-04-02T03:14:32.574 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:32.577784 I | ceph-csi: created kubernetes csi secrets for cluster "rook-ceph" 2022-04-02T03:14:32.575 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:32.577811 I | cephclient: getting or creating ceph auth key "client.crash" 2022-04-02T03:14:32.999 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:33.002030 I | ceph-crashcollector-controller: created kubernetes crash collector secret for cluster "rook-ceph" 2022-04-02T03:14:32.999 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:33.002080 I | cephclient: create rbd-mirror bootstrap peer token "client.rbd-mirror-peer" 2022-04-02T03:14:33.000 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:33.002089 I | cephclient: getting or creating ceph auth key "client.rbd-mirror-peer" 2022-04-02T03:14:33.445 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:33.448894 I | cephclient: successfully created rbd-mirror bootstrap peer token for cluster "rook-ceph" 2022-04-02T03:14:33.478 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:33.481745 I | op-mgr: start running mgr 2022-04-02T03:14:33.478 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:33.481792 I | cephclient: getting or creating ceph auth key "mgr.a" 2022-04-02T03:14:33.941 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:33.944600 I | op-mgr: deployment for mgr rook-ceph-mgr-a already exists. updating if needed 2022-04-02T03:14:33.951 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:33.955122 I | op-k8sutil: deployment "rook-ceph-mgr-a" did not change, nothing to update 2022-04-02T03:14:33.955 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:33.959102 I | op-mgr: setting services to point to mgr "a" 2022-04-02T03:14:34.552 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:34.555799 I | op-mgr: no need to update service "rook-ceph-mgr" 2022-04-02T03:14:34.553 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:34.555943 I | op-mgr: successful modules: balancer 2022-04-02T03:14:34.567 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:34.570277 I | op-osd: start running osds in namespace "rook-ceph" 2022-04-02T03:14:34.567 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:34.570326 I | op-osd: wait timeout for healthy OSDs during upgrade or restart is "10m0s" 2022-04-02T03:14:34.637 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:34.641167 I | op-osd: start provisioning the OSDs on PVCs, if needed 2022-04-02T03:14:34.750 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:34.753821 I | op-osd: creating 1 new PVCs for device set "smithi029-lv-1" 2022-04-02T03:14:34.964 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:34.967844 I | op-osd: successfully provisioned PVC "smithi029-lv-1-data-0xzkjz" 2022-04-02T03:14:34.964 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:34.967863 I | op-osd: creating 1 new PVCs for device set "smithi029-lv-2" 2022-04-02T03:14:35.152 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:35.155892 I | op-osd: successfully provisioned PVC "smithi029-lv-2-data-084vgd" 2022-04-02T03:14:35.240 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:35.155926 I | op-osd: creating 1 new PVCs for device set "smithi029-lv-3" 2022-04-02T03:14:35.369 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:35.372519 I | op-osd: successfully provisioned PVC "smithi029-lv-3-data-0f2zlj" 2022-04-02T03:14:35.369 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:35.372552 I | op-osd: creating 1 new PVCs for device set "smithi029-lv-4" 2022-04-02T03:14:35.553 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:35.556076 I | op-osd: successfully provisioned PVC "smithi029-lv-4-data-0sxfm8" 2022-04-02T03:14:35.553 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:35.556107 I | op-osd: creating 1 new PVCs for device set "smithi104-lv-1" 2022-04-02T03:14:35.751 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:35.754659 I | op-osd: successfully provisioned PVC "smithi104-lv-1-data-0jfbpq" 2022-04-02T03:14:35.751 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:35.754690 I | op-osd: creating 1 new PVCs for device set "smithi104-lv-2" 2022-04-02T03:14:35.884 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:35.887681 I | op-mgr: successful modules: prometheus 2022-04-02T03:14:35.887 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:35.889623 I | op-mgr: successful modules: dashboard 2022-04-02T03:14:35.887 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:35.890251 I | op-mgr: successful modules: mgr module(s) from the spec 2022-04-02T03:14:35.985 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:35.988971 I | op-osd: successfully provisioned PVC "smithi104-lv-2-data-08rssq" 2022-04-02T03:14:35.986 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:35.989018 I | op-osd: creating 1 new PVCs for device set "smithi104-lv-3" 2022-04-02T03:14:36.153 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:36.156242 I | op-osd: successfully provisioned PVC "smithi104-lv-3-data-0jlrxt" 2022-04-02T03:14:36.153 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:36.156284 I | op-osd: creating 1 new PVCs for device set "smithi104-lv-4" 2022-04-02T03:14:36.353 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:36.356519 I | op-osd: successfully provisioned PVC "smithi104-lv-4-data-0nmqwk" 2022-04-02T03:14:36.353 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:36.356554 I | op-osd: creating 1 new PVCs for device set "smithi142-lv-1" 2022-04-02T03:14:36.553 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:36.556299 I | op-osd: successfully provisioned PVC "smithi142-lv-1-data-08v2jt" 2022-04-02T03:14:36.553 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:36.556336 I | op-osd: creating 1 new PVCs for device set "smithi142-lv-2" 2022-04-02T03:14:36.752 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:36.755683 I | op-osd: successfully provisioned PVC "smithi142-lv-2-data-0qdbgg" 2022-04-02T03:14:36.752 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:36.755713 I | op-osd: creating 1 new PVCs for device set "smithi142-lv-3" 2022-04-02T03:14:36.985 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:36.988930 I | op-osd: successfully provisioned PVC "smithi142-lv-3-data-05pxj5" 2022-04-02T03:14:36.986 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:36.988967 I | op-osd: creating 1 new PVCs for device set "smithi142-lv-4" 2022-04-02T03:14:37.134 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph osd stat -f json 2022-04-02T03:14:37.220 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:37.223486 I | op-osd: successfully provisioned PVC "smithi142-lv-4-data-0hs4jx" 2022-04-02T03:14:37.258 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:37.261960 I | op-osd: OSD will have its main bluestore block on "smithi029-lv-1-data-0xzkjz" 2022-04-02T03:14:37.465 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:37.468188 I | op-mgr: successful modules: orchestrator modules 2022-04-02T03:14:37.563 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:37.566506 I | op-osd: started OSD provisioning job for PVC "smithi029-lv-1-data-0xzkjz" 2022-04-02T03:14:37.563 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:37.566609 I | op-osd: OSD will have its main bluestore block on "smithi029-lv-2-data-084vgd" 2022-04-02T03:14:37.767 INFO:teuthology.orchestra.run.smithi029.stdout: 2022-04-02T03:14:37.786 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":2,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2022-04-02T03:14:37.787 INFO:tasks.rook: have 0/12 OSDs 2022-04-02T03:14:37.965 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:37.968366 I | op-osd: started OSD provisioning job for PVC "smithi029-lv-2-data-084vgd" 2022-04-02T03:14:37.965 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:37.968402 I | op-osd: OSD will have its main bluestore block on "smithi029-lv-3-data-0f2zlj" 2022-04-02T03:14:39.113 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:39.116881 I | op-osd: started OSD provisioning job for PVC "smithi029-lv-3-data-0f2zlj" 2022-04-02T03:14:39.114 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:39.116919 I | op-osd: OSD will have its main bluestore block on "smithi029-lv-4-data-0sxfm8" 2022-04-02T03:14:40.071 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:40.074263 I | op-osd: started OSD provisioning job for PVC "smithi029-lv-4-data-0sxfm8" 2022-04-02T03:14:40.071 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:40.074301 I | op-osd: OSD will have its main bluestore block on "smithi104-lv-1-data-0jfbpq" 2022-04-02T03:14:40.912 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:40.915987 I | op-osd: started OSD provisioning job for PVC "smithi104-lv-1-data-0jfbpq" 2022-04-02T03:14:40.913 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:40.916020 I | op-osd: OSD will have its main bluestore block on "smithi104-lv-2-data-08rssq" 2022-04-02T03:14:42.338 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:42.341433 I | op-osd: started OSD provisioning job for PVC "smithi104-lv-2-data-08rssq" 2022-04-02T03:14:42.338 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:42.341474 I | op-osd: OSD will have its main bluestore block on "smithi104-lv-3-data-0jlrxt" 2022-04-02T03:14:43.598 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:43.601653 I | op-osd: started OSD provisioning job for PVC "smithi104-lv-3-data-0jlrxt" 2022-04-02T03:14:43.598 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:43.601691 I | op-osd: OSD will have its main bluestore block on "smithi104-lv-4-data-0nmqwk" 2022-04-02T03:14:44.431 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:44.434651 I | op-osd: started OSD provisioning job for PVC "smithi104-lv-4-data-0nmqwk" 2022-04-02T03:14:44.431 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:44.434689 I | op-osd: OSD will have its main bluestore block on "smithi142-lv-1-data-08v2jt" 2022-04-02T03:14:45.379 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:45.382969 I | op-osd: started OSD provisioning job for PVC "smithi142-lv-1-data-08v2jt" 2022-04-02T03:14:45.380 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:45.383009 I | op-osd: OSD will have its main bluestore block on "smithi142-lv-2-data-0qdbgg" 2022-04-02T03:14:46.513 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:46.516997 I | op-osd: started OSD provisioning job for PVC "smithi142-lv-2-data-0qdbgg" 2022-04-02T03:14:46.514 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:46.517039 I | op-osd: OSD will have its main bluestore block on "smithi142-lv-3-data-05pxj5" 2022-04-02T03:14:47.738 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:47.741983 I | op-osd: started OSD provisioning job for PVC "smithi142-lv-3-data-05pxj5" 2022-04-02T03:14:47.739 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:47.742022 I | op-osd: OSD will have its main bluestore block on "smithi142-lv-4-data-0hs4jx" 2022-04-02T03:14:47.788 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph osd stat -f json 2022-04-02T03:14:49.747 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:49.750188 I | op-osd: started OSD provisioning job for PVC "smithi142-lv-4-data-0hs4jx" 2022-04-02T03:14:49.747 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:49.750218 I | op-osd: start provisioning the OSDs on nodes, if needed 2022-04-02T03:14:49.747 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:49.750224 I | op-osd: no nodes are defined for configuring OSDs on raw devices 2022-04-02T03:14:49.751 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:49.754378 I | op-osd: OSD orchestration status for PVC smithi029-lv-1-data-0xzkjz is "starting" 2022-04-02T03:14:49.751 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:49.754418 I | op-osd: OSD orchestration status for PVC smithi029-lv-2-data-084vgd is "starting" 2022-04-02T03:14:49.751 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:49.754442 I | op-osd: OSD orchestration status for PVC smithi029-lv-3-data-0f2zlj is "starting" 2022-04-02T03:14:49.751 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:49.754464 I | op-osd: OSD orchestration status for PVC smithi029-lv-4-data-0sxfm8 is "starting" 2022-04-02T03:14:49.752 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:49.754487 I | op-osd: OSD orchestration status for PVC smithi104-lv-1-data-0jfbpq is "starting" 2022-04-02T03:14:49.752 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:49.754509 I | op-osd: OSD orchestration status for PVC smithi104-lv-2-data-08rssq is "starting" 2022-04-02T03:14:49.752 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:49.754537 I | op-osd: OSD orchestration status for PVC smithi104-lv-3-data-0jlrxt is "starting" 2022-04-02T03:14:49.752 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:49.754571 I | op-osd: OSD orchestration status for PVC smithi104-lv-4-data-0nmqwk is "starting" 2022-04-02T03:14:49.754 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:49.754589 I | op-osd: OSD orchestration status for PVC smithi142-lv-1-data-08v2jt is "starting" 2022-04-02T03:14:49.754 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:49.754602 I | op-osd: OSD orchestration status for PVC smithi142-lv-2-data-0qdbgg is "starting" 2022-04-02T03:14:49.755 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:49.754616 I | op-osd: OSD orchestration status for PVC smithi142-lv-3-data-05pxj5 is "starting" 2022-04-02T03:14:49.755 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:49.754633 I | op-osd: OSD orchestration status for PVC smithi142-lv-4-data-0hs4jx is "starting" 2022-04-02T03:14:50.153 INFO:teuthology.orchestra.run.smithi029.stdout: 2022-04-02T03:14:50.177 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":2,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2022-04-02T03:14:50.178 INFO:tasks.rook: have 0/12 OSDs 2022-04-02T03:14:58.267 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:58.270735 I | op-osd: OSD orchestration status for node smithi104-lv-1-data-0jfbpq is "orchestrating" 2022-04-02T03:14:59.982 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:14:59.986573 I | op-osd: OSD orchestration status for PVC smithi104-lv-1-data-0jfbpq is "orchestrating" 2022-04-02T03:15:00.072 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:15:00.076022 I | op-osd: OSD orchestration status for node smithi104-lv-2-data-08rssq is "orchestrating" 2022-04-02T03:15:00.178 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph osd stat -f json 2022-04-02T03:15:00.849 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:15:00.852199 I | op-osd: OSD orchestration status for PVC smithi104-lv-2-data-08rssq is "orchestrating" 2022-04-02T03:15:01.577 INFO:teuthology.orchestra.run.smithi029.stdout: 2022-04-02T03:15:01.607 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":2,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2022-04-02T03:15:01.608 INFO:tasks.rook: have 0/12 OSDs 2022-04-02T03:15:02.253 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:15:02.257600 I | op-osd: OSD orchestration status for node smithi029-lv-1-data-0xzkjz is "orchestrating" 2022-04-02T03:15:02.616 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:15:02.619943 I | op-osd: OSD orchestration status for node smithi104-lv-3-data-0jlrxt is "orchestrating" 2022-04-02T03:15:02.619 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:15:02.620245 I | op-osd: OSD orchestration status for node smithi104-lv-4-data-0nmqwk is "orchestrating" 2022-04-02T03:15:02.689 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:15:02.693424 I | op-osd: OSD orchestration status for node smithi142-lv-3-data-05pxj5 is "orchestrating" 2022-04-02T03:15:02.858 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:15:02.862199 I | op-osd: OSD orchestration status for node smithi142-lv-2-data-0qdbgg is "orchestrating" 2022-04-02T03:15:03.809 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:15:03.811968 I | op-osd: OSD orchestration status for PVC smithi104-lv-4-data-0nmqwk is "orchestrating" 2022-04-02T03:15:03.810 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:15:03.812144 I | op-osd: OSD orchestration status for PVC smithi029-lv-1-data-0xzkjz is "orchestrating" 2022-04-02T03:15:03.810 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:15:03.812279 I | op-osd: OSD orchestration status for PVC smithi104-lv-3-data-0jlrxt is "orchestrating" 2022-04-02T03:15:03.834 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:15:03.836555 I | op-osd: OSD orchestration status for PVC smithi142-lv-3-data-05pxj5 is "orchestrating" 2022-04-02T03:15:03.834 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:15:03.836597 I | op-osd: OSD orchestration status for PVC smithi142-lv-2-data-0qdbgg is "orchestrating" 2022-04-02T03:15:04.374 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:15:04.378534 I | op-osd: OSD orchestration status for node smithi142-lv-1-data-08v2jt is "orchestrating" 2022-04-02T03:15:04.753 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:15:04.757425 I | op-osd: OSD orchestration status for node smithi029-lv-2-data-084vgd is "orchestrating" 2022-04-02T03:15:05.729 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:15:05.732110 I | op-osd: OSD orchestration status for PVC smithi142-lv-1-data-08v2jt is "orchestrating" 2022-04-02T03:15:05.735 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:15:05.734243 I | op-osd: OSD orchestration status for PVC smithi104-lv-3-data-0jlrxt is "failed" 2022-04-02T03:15:05.736 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:15:05.734302 E | op-osd: failed to provision OSD(s) on PVC smithi104-lv-3-data-0jlrxt. &{OSDs:[] Status:failed PvcBackedOSD:true Message:failed to configure devices: failed to initialize devices on PVC: failed to run ceph-volume. Running command: /usr/bin/ceph-authtool --gen-print-key 2022-04-02T03:15:05.736 INFO:tasks.rook.operator.smithi029.stdout:Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new f2994011-a714-47a8-a17d-b0dbbffec787 2022-04-02T03:15:05.736 INFO:tasks.rook.operator.smithi029.stdout: stderr: unable to get monitor info from DNS SRV with service name: ceph-mon 2022-04-02T03:15:05.736 INFO:tasks.rook.operator.smithi029.stdout: stderr: 2022-04-02T03:15:05.071+0000 7fd488959700 -1 failed for service _ceph-mon._tcp 2022-04-02T03:15:05.737 INFO:tasks.rook.operator.smithi029.stdout: stderr: 2022-04-02T03:15:05.071+0000 7fd488959700 -1 monclient: get_monmap_and_config cannot identify monitors to contact 2022-04-02T03:15:05.737 INFO:tasks.rook.operator.smithi029.stdout: stderr: [errno 2] RADOS object not found (error connecting to the cluster) 2022-04-02T03:15:05.739 INFO:tasks.rook.operator.smithi029.stdout:Traceback (most recent call last): 2022-04-02T03:15:05.739 INFO:tasks.rook.operator.smithi029.stdout: File "/usr/sbin/ceph-volume", line 11, in 2022-04-02T03:15:05.740 INFO:tasks.rook.operator.smithi029.stdout: load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2022-04-02T03:15:05.740 INFO:tasks.rook.operator.smithi029.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2022-04-02T03:15:05.740 INFO:tasks.rook.operator.smithi029.stdout: self.main(self.argv) 2022-04-02T03:15:05.740 INFO:tasks.rook.operator.smithi029.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2022-04-02T03:15:05.740 INFO:tasks.rook.operator.smithi029.stdout: return f(*a, **kw) 2022-04-02T03:15:05.741 INFO:tasks.rook.operator.smithi029.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2022-04-02T03:15:05.741 INFO:tasks.rook.operator.smithi029.stdout: terminal.dispatch(self.mapper, subcommand_args) 2022-04-02T03:15:05.741 INFO:tasks.rook.operator.smithi029.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2022-04-02T03:15:05.741 INFO:tasks.rook.operator.smithi029.stdout: instance.main() 2022-04-02T03:15:05.742 INFO:tasks.rook.operator.smithi029.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/main.py", line 32, in main 2022-04-02T03:15:05.742 INFO:tasks.rook.operator.smithi029.stdout: terminal.dispatch(self.mapper, self.argv) 2022-04-02T03:15:05.742 INFO:tasks.rook.operator.smithi029.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2022-04-02T03:15:05.742 INFO:tasks.rook.operator.smithi029.stdout: instance.main() 2022-04-02T03:15:05.742 INFO:tasks.rook.operator.smithi029.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 169, in main 2022-04-02T03:15:05.743 INFO:tasks.rook.operator.smithi029.stdout: self.safe_prepare(self.args) 2022-04-02T03:15:05.743 INFO:tasks.rook.operator.smithi029.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 91, in safe_prepare 2022-04-02T03:15:05.743 INFO:tasks.rook.operator.smithi029.stdout: self.prepare() 2022-04-02T03:15:05.743 INFO:tasks.rook.operator.smithi029.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2022-04-02T03:15:05.743 INFO:tasks.rook.operator.smithi029.stdout: return func(*a, **kw) 2022-04-02T03:15:05.744 INFO:tasks.rook.operator.smithi029.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 125, in prepare 2022-04-02T03:15:05.748 INFO:tasks.rook.operator.smithi029.stdout: osd_fsid, json.dumps(secrets)) 2022-04-02T03:15:05.749 INFO:tasks.rook.operator.smithi029.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/util/prepare.py", line 173, in create_id 2022-04-02T03:15:05.749 INFO:tasks.rook.operator.smithi029.stdout: raise RuntimeError('Unable to create a new OSD id') 2022-04-02T03:15:05.749 INFO:tasks.rook.operator.smithi029.stdout:RuntimeError: Unable to create a new OSD id. debug logs below: 2022-04-02T03:15:05.749 INFO:tasks.rook.operator.smithi029.stdout:[2022-04-02 03:15:04,570][ceph_volume.main][INFO ] Running command: ceph-volume --log-path /var/log/ceph/smithi104-lv-3-data-0jlrxt raw prepare --bluestore --data /dev/mapper/vg_nvme-lv_3 2022-04-02T03:15:05.750 INFO:tasks.rook.operator.smithi029.stdout:[2022-04-02 03:15:04,571][ceph_volume.process][INFO ] Running command: /usr/bin/lsblk -plno KNAME,NAME,TYPE 2022-04-02T03:15:05.750 INFO:tasks.rook.operator.smithi029.stdout:[2022-04-02 03:15:04,581][ceph_volume.process][INFO ] stdout /dev/loop0 /dev/loop0 loop 2022-04-02T03:15:05.750 INFO:tasks.rook.operator.smithi029.stdout:[2022-04-02 03:15:04,582][ceph_volume.process][INFO ] stdout /dev/loop1 /dev/loop1 loop 2022-04-02T03:15:05.750 INFO:tasks.rook.operator.smithi029.stdout:[2022-04-02 03:15:04,582][ceph_volume.process][INFO ] stdout /dev/loop2 /dev/loop2 loop 2022-04-02T03:15:05.750 INFO:tasks.rook.operator.smithi029.stdout:[2022-04-02 03:15:04,582][ceph_volume.process][INFO ] stdout /dev/loop3 /dev/loop3 loop 2022-04-02T03:15:05.751 INFO:tasks.rook.operator.smithi029.stdout:[2022-04-02 03:15:04,582][ceph_volume.process][INFO ] stdout /dev/sda /dev/sda disk 2022-04-02T03:15:05.751 INFO:tasks.rook.operator.smithi029.stdout:[2022-04-02 03:15:04,582][ceph_volume.process][INFO ] stdout /dev/sda1 /dev/sda1 part 2022-04-02T03:15:05.751 INFO:tasks.rook.operator.smithi029.stdout:[2022-04-02 03:15:04,582][ceph_volume.process][INFO ] stdout /dev/dm-0 /dev/mapper/vg_nvme-lv_1 lvm 2022-04-02T03:15:05.751 INFO:tasks.rook.operator.smithi029.stdout:[2022-04-02 03:15:04,582][ceph_volume.process][INFO ] stdout /dev/dm-1 /dev/mapper/vg_nvme-lv_2 lvm 2022-04-02T03:15:05.752 INFO:tasks.rook.operator.smithi029.stdout:[2022-04-02 03:15:04,582][ceph_volume.process][INFO ] stdout /dev/dm-2 /dev/mapper/vg_nvme-lv_3 lvm 2022-04-02T03:15:05.752 INFO:tasks.rook.operator.smithi029.stdout:[2022-04-02 03:15:04,582][ceph_volume.process][INFO ] stdout /dev/dm-3 /dev/mapper/vg_nvme-lv_4 lvm 2022-04-02T03:15:05.752 INFO:tasks.rook.operator.smithi029.stdout:[2022-04-02 03:15:04,582][ceph_volume.process][INFO ] stdout /dev/dm-4 /dev/mapper/vg_nvme-lv_5 lvm 2022-04-02T03:15:05.752 INFO:tasks.rook.operator.smithi029.stdout:[2022-04-02 03:15:04,582][ceph_volume.process][INFO ] stdout /dev/nvme0n1 /dev/nvme0n1 disk 2022-04-02T03:15:05.752 INFO:tasks.rook.operator.smithi029.stdout:[2022-04-02 03:15:04,582][ceph_volume.process][INFO ] stdout /dev/nvme1n1 /dev/nvme1n1 disk 2022-04-02T03:15:05.753 INFO:tasks.rook.operator.smithi029.stdout:[2022-04-02 03:15:04,582][ceph_volume.process][INFO ] stdout /dev/nvme2n1 /dev/nvme2n1 disk 2022-04-02T03:15:05.753 INFO:tasks.rook.operator.smithi029.stdout:[2022-04-02 03:15:04,582][ceph_volume.process][INFO ] stdout /dev/nvme3n1 /dev/nvme3n1 disk 2022-04-02T03:15:05.753 INFO:tasks.rook.operator.smithi029.stdout:[2022-04-02 03:15:04,583][ceph_volume.process][INFO ] stdout /dev/nvme4n1 /dev/nvme4n1 disk 2022-04-02T03:15:05.753 INFO:tasks.rook.operator.smithi029.stdout:[2022-04-02 03:15:04,596][ceph_volume.util.system][WARNING] Executable lvs not found on the host, will return lvs as-is 2022-04-02T03:15:05.753 INFO:tasks.rook.operator.smithi029.stdout:[2022-04-02 03:15:04,596][ceph_volume.process][INFO ] Running command: lvs --noheadings --readonly --separator=";" -a --units=b --nosuffix -S lv_path=/dev/mapper/vg_nvme-lv_3 -o lv_tags,lv_path,lv_name,vg_name,lv_uuid,lv_size 2022-04-02T03:15:05.754 INFO:tasks.rook.operator.smithi029.stdout:[2022-04-02 03:15:04,910][ceph_volume.process][INFO ] Running command: /usr/bin/lsblk --nodeps -P -o NAME,KNAME,MAJ:MIN,FSTYPE,MOUNTPOINT,LABEL,UUID,RO,RM,MODEL,SIZE,STATE,OWNER,GROUP,MODE,ALIGNMENT,PHY-SEC,LOG-SEC,ROTA,SCHED,TYPE,DISC-ALN,DISC-GRAN,DISC-MAX,DISC-ZERO,PKNAME,PARTLABEL /dev/mapper/vg_nvme-lv_3 2022-04-02T03:15:05.754 INFO:tasks.rook.operator.smithi029.stdout:[2022-04-02 03:15:04,915][ceph_volume.process][INFO ] stdout NAME="vg_nvme-lv_3" KNAME="dm-2" MAJ:MIN="253:2" FSTYPE="" MOUNTPOINT="" LABEL="" UUID="" RO="0" RM="0" MODEL="" SIZE="89.4G" STATE="running" OWNER="root" GROUP="disk" MODE="brw-rw----" ALIGNMENT="0" PHY-SEC="512" LOG-SEC="512" ROTA="0" SCHED="" TYPE="lvm" DISC-ALN="0" DISC-GRAN="512B" DISC-MAX="2T" DISC-ZERO="0" PKNAME="" PARTLABEL="" 2022-04-02T03:15:05.754 INFO:tasks.rook.operator.smithi029.stdout:[2022-04-02 03:15:04,916][ceph_volume.process][INFO ] Running command: /usr/sbin/blkid -c /dev/null -p /dev/mapper/vg_nvme-lv_3 2022-04-02T03:15:05.754 INFO:tasks.rook.operator.smithi029.stdout:[2022-04-02 03:15:04,919][ceph_volume.util.disk][INFO ] opening device /dev/mapper/vg_nvme-lv_3 to check for BlueStore label 2022-04-02T03:15:05.755 INFO:tasks.rook.operator.smithi029.stdout:[2022-04-02 03:15:04,919][ceph_volume.util.disk][INFO ] opening device /dev/mapper/vg_nvme-lv_3 to check for BlueStore label 2022-04-02T03:15:05.755 INFO:tasks.rook.operator.smithi029.stdout:[2022-04-02 03:15:04,920][ceph_volume.process][INFO ] Running command: /usr/sbin/udevadm info --query=property /dev/mapper/vg_nvme-lv_3 2022-04-02T03:15:05.755 INFO:tasks.rook.operator.smithi029.stdout:[2022-04-02 03:15:04,926][ceph_volume.process][INFO ] stdout DEVLINKS=/dev/mapper/vg_nvme-lv_3 /dev/vg_nvme/lv_3 /dev/disk/by-id/dm-name-vg_nvme-lv_3 /dev/disk/by-id/dm-uuid-LVM-nNFcexrtq2jlNGKS14RaFdSTNQtJ1XS0lpWXInTLFSJNsgbq4ZZaUEAD8n2StRU9 2022-04-02T03:15:05.755 INFO:tasks.rook.operator.smithi029.stdout:[2022-04-02 03:15:04,927][ceph_volume.process][INFO ] stdout DEVNAME=/dev/dm-2 2022-04-02T03:15:05.755 INFO:tasks.rook.operator.smithi029.stdout:[2022-04-02 03:15:04,927][ceph_volume.process][INFO ] stdout DEVPATH=/devices/virtual/block/dm-2 2022-04-02T03:15:05.756 INFO:tasks.rook.operator.smithi029.stdout:[2022-04-02 03:15:04,927][ceph_volume.process][INFO ] stdout DEVTYPE=disk 2022-04-02T03:15:05.756 INFO:tasks.rook.operator.smithi029.stdout:[2022-04-02 03:15:04,927][ceph_volume.process][INFO ] stdout DM_LV_NAME=lv_3 2022-04-02T03:15:05.756 INFO:tasks.rook.operator.smithi029.stdout:[2022-04-02 03:15:04,927][ceph_volume.process][INFO ] stdout DM_NAME=vg_nvme-lv_3 2022-04-02T03:15:05.756 INFO:tasks.rook.operator.smithi029.stdout:[2022-04-02 03:15:04,927][ceph_volume.process][INFO ] stdout DM_SUSPENDED=0 2022-04-02T03:15:05.756 INFO:tasks.rook.operator.smithi029.stdout:[2022-04-02 03:15:04,927][ceph_volume.process][INFO ] stdout DM_UDEV_DISABLE_LIBRARY_FALLBACK_FLAG=1 2022-04-02T03:15:05.757 INFO:tasks.rook.operator.smithi029.stdout:[2022-04-02 03:15:04,927][ceph_volume.process][INFO ] stdout DM_UDEV_PRIMARY_SOURCE_FLAG=1 2022-04-02T03:15:05.757 INFO:tasks.rook.operator.smithi029.stdout:[2022-04-02 03:15:04,927][ceph_volume.process][INFO ] stdout DM_UDEV_RULES=1 2022-04-02T03:15:05.757 INFO:tasks.rook.operator.smithi029.stdout:[2022-04-02 03:15:04,927][ceph_volume.process][INFO ] stdout DM_UDEV_RULES_VSN=2 2022-04-02T03:15:05.757 INFO:tasks.rook.operator.smithi029.stdout:[2022-04-02 03:15:04,927][ceph_volume.process][INFO ] stdout DM_UUID=LVM-nNFcexrtq2jlNGKS14RaFdSTNQtJ1XS0lpWXInTLFSJNsgbq4ZZaUEAD8n2StRU9 2022-04-02T03:15:05.757 INFO:tasks.rook.operator.smithi029.stdout:[2022-04-02 03:15:04,927][ceph_volume.process][INFO ] stdout DM_VG_NAME=vg_nvme 2022-04-02T03:15:05.758 INFO:tasks.rook.operator.smithi029.stdout:[2022-04-02 03:15:04,927][ceph_volume.process][INFO ] stdout MAJOR=253 2022-04-02T03:15:05.758 INFO:tasks.rook.operator.smithi029.stdout:[2022-04-02 03:15:04,927][ceph_volume.process][INFO ] stdout MINOR=2 2022-04-02T03:15:05.758 INFO:tasks.rook.operator.smithi029.stdout:[2022-04-02 03:15:04,927][ceph_volume.process][INFO ] stdout SUBSYSTEM=block 2022-04-02T03:15:05.758 INFO:tasks.rook.operator.smithi029.stdout:[2022-04-02 03:15:04,927][ceph_volume.process][INFO ] stdout TAGS=:systemd: 2022-04-02T03:15:05.758 INFO:tasks.rook.operator.smithi029.stdout:[2022-04-02 03:15:04,927][ceph_volume.process][INFO ] stdout USEC_INITIALIZED=227059161 2022-04-02T03:15:05.759 INFO:tasks.rook.operator.smithi029.stdout:[2022-04-02 03:15:04,928][ceph_volume.process][INFO ] Running command: /usr/bin/ceph-authtool --gen-print-key 2022-04-02T03:15:05.759 INFO:tasks.rook.operator.smithi029.stdout:[2022-04-02 03:15:04,944][ceph_volume.process][INFO ] stdout AQC4v0diJP88OBAARkHyrc1ioE7ljbCaDa1trQ== 2022-04-02T03:15:05.759 INFO:tasks.rook.operator.smithi029.stdout:[2022-04-02 03:15:04,945][ceph_volume.process][INFO ] Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new f2994011-a714-47a8-a17d-b0dbbffec787 2022-04-02T03:15:05.759 INFO:tasks.rook.operator.smithi029.stdout:[2022-04-02 03:15:05,077][ceph_volume.process][INFO ] stderr unable to get monitor info from DNS SRV with service name: ceph-mon 2022-04-02T03:15:05.760 INFO:tasks.rook.operator.smithi029.stdout:[2022-04-02 03:15:05,078][ceph_volume.process][INFO ] stderr 2022-04-02T03:15:05.071+0000 7fd488959700 -1 failed for service _ceph-mon._tcp 2022-04-02T03:15:05.760 INFO:tasks.rook.operator.smithi029.stdout:[2022-04-02 03:15:05,078][ceph_volume.process][INFO ] stderr 2022-04-02T03:15:05.071+0000 7fd488959700 -1 monclient: get_monmap_and_config cannot identify monitors to contact 2022-04-02T03:15:05.760 INFO:tasks.rook.operator.smithi029.stdout:[2022-04-02 03:15:05,078][ceph_volume.process][INFO ] stderr [errno 2] RADOS object not found (error connecting to the cluster) 2022-04-02T03:15:05.760 INFO:tasks.rook.operator.smithi029.stdout:[2022-04-02 03:15:05,078][ceph_volume.devices.raw.prepare][ERROR ] raw prepare was unable to complete 2022-04-02T03:15:05.760 INFO:tasks.rook.operator.smithi029.stdout:Traceback (most recent call last): 2022-04-02T03:15:05.761 INFO:tasks.rook.operator.smithi029.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 91, in safe_prepare 2022-04-02T03:15:05.761 INFO:tasks.rook.operator.smithi029.stdout: self.prepare() 2022-04-02T03:15:05.761 INFO:tasks.rook.operator.smithi029.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2022-04-02T03:15:05.761 INFO:tasks.rook.operator.smithi029.stdout: return func(*a, **kw) 2022-04-02T03:15:05.761 INFO:tasks.rook.operator.smithi029.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 125, in prepare 2022-04-02T03:15:05.762 INFO:tasks.rook.operator.smithi029.stdout: osd_fsid, json.dumps(secrets)) 2022-04-02T03:15:05.762 INFO:tasks.rook.operator.smithi029.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/util/prepare.py", line 173, in create_id 2022-04-02T03:15:05.762 INFO:tasks.rook.operator.smithi029.stdout: raise RuntimeError('Unable to create a new OSD id') 2022-04-02T03:15:05.762 INFO:tasks.rook.operator.smithi029.stdout:RuntimeError: Unable to create a new OSD id 2022-04-02T03:15:05.762 INFO:tasks.rook.operator.smithi029.stdout:[2022-04-02 03:15:05,078][ceph_volume.devices.raw.prepare][INFO ] will rollback OSD ID creation 2022-04-02T03:15:05.763 INFO:tasks.rook.operator.smithi029.stdout:[2022-04-02 03:15:05,079][ceph_volume][ERROR ] exception caught by decorator 2022-04-02T03:15:05.763 INFO:tasks.rook.operator.smithi029.stdout:Traceback (most recent call last): 2022-04-02T03:15:05.763 INFO:tasks.rook.operator.smithi029.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2022-04-02T03:15:05.763 INFO:tasks.rook.operator.smithi029.stdout: return f(*a, **kw) 2022-04-02T03:15:05.763 INFO:tasks.rook.operator.smithi029.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2022-04-02T03:15:05.764 INFO:tasks.rook.operator.smithi029.stdout: terminal.dispatch(self.mapper, subcommand_args) 2022-04-02T03:15:05.764 INFO:tasks.rook.operator.smithi029.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2022-04-02T03:15:05.764 INFO:tasks.rook.operator.smithi029.stdout: instance.main() 2022-04-02T03:15:05.764 INFO:tasks.rook.operator.smithi029.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/main.py", line 32, in main 2022-04-02T03:15:05.764 INFO:tasks.rook.operator.smithi029.stdout: terminal.dispatch(self.mapper, self.argv) 2022-04-02T03:15:05.765 INFO:tasks.rook.operator.smithi029.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2022-04-02T03:15:05.765 INFO:tasks.rook.operator.smithi029.stdout: instance.main() 2022-04-02T03:15:05.765 INFO:tasks.rook.operator.smithi029.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 169, in main 2022-04-02T03:15:05.765 INFO:tasks.rook.operator.smithi029.stdout: self.safe_prepare(self.args) 2022-04-02T03:15:05.766 INFO:tasks.rook.operator.smithi029.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 91, in safe_prepare 2022-04-02T03:15:05.766 INFO:tasks.rook.operator.smithi029.stdout: self.prepare() 2022-04-02T03:15:05.766 INFO:tasks.rook.operator.smithi029.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2022-04-02T03:15:05.766 INFO:tasks.rook.operator.smithi029.stdout: return func(*a, **kw) 2022-04-02T03:15:05.766 INFO:tasks.rook.operator.smithi029.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 125, in prepare 2022-04-02T03:15:05.767 INFO:tasks.rook.operator.smithi029.stdout: osd_fsid, json.dumps(secrets)) 2022-04-02T03:15:05.767 INFO:tasks.rook.operator.smithi029.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/util/prepare.py", line 173, in create_id 2022-04-02T03:15:05.767 INFO:tasks.rook.operator.smithi029.stdout: raise RuntimeError('Unable to create a new OSD id') 2022-04-02T03:15:05.767 INFO:tasks.rook.operator.smithi029.stdout:RuntimeError: Unable to create a new OSD id 2022-04-02T03:15:05.767 INFO:tasks.rook.operator.smithi029.stdout:: exit status 1} 2022-04-02T03:15:05.768 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:15:05.750962 I | op-osd: OSD orchestration status for PVC smithi029-lv-2-data-084vgd is "orchestrating" 2022-04-02T03:15:05.790 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:15:05.794342 I | op-osd: OSD orchestration status for node smithi142-lv-4-data-0hs4jx is "orchestrating" 2022-04-02T03:15:05.790 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:15:05.794384 I | op-osd: OSD orchestration status for node smithi029-lv-3-data-0f2zlj is "orchestrating" 2022-04-02T03:15:06.531 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:15:06.535654 I | op-osd: OSD orchestration status for PVC smithi142-lv-4-data-0hs4jx is "orchestrating" 2022-04-02T03:15:06.532 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:15:06.535781 I | op-osd: OSD orchestration status for PVC smithi029-lv-3-data-0f2zlj is "orchestrating" 2022-04-02T03:15:06.618 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:15:06.622156 I | op-osd: OSD orchestration status for PVC smithi104-lv-1-data-0jfbpq is "completed" 2022-04-02T03:15:06.619 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:15:06.622190 I | op-osd: creating OSD 0 on PVC "smithi104-lv-1-data-0jfbpq" 2022-04-02T03:15:06.619 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:15:06.622210 I | op-osd: OSD will have its main bluestore block on "smithi104-lv-1-data-0jfbpq" 2022-04-02T03:15:06.777 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:15:06.781839 E | ceph-spec: failed to update cluster condition to {Type:Ready Status:True Reason:ClusterCreated Message:Cluster created successfully LastHeartbeatTime:2022-04-02 03:15:06.753184293 +0000 UTC m=+364.541081105 LastTransitionTime:2022-04-02 03:13:42 +0000 UTC}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-04-02T03:15:06.911 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:15:06.915479 I | op-osd: OSD orchestration status for node smithi029-lv-4-data-0sxfm8 is "orchestrating" 2022-04-02T03:15:07.304 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:15:07.308878 I | op-osd: OSD orchestration status for PVC smithi029-lv-4-data-0sxfm8 is "orchestrating" 2022-04-02T03:15:08.198 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:15:08.201971 I | op-osd: OSD orchestration status for PVC smithi104-lv-2-data-08rssq is "completed" 2022-04-02T03:15:08.199 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:15:08.202001 I | op-osd: creating OSD 1 on PVC "smithi104-lv-2-data-08rssq" 2022-04-02T03:15:08.199 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:15:08.202013 I | op-osd: OSD will have its main bluestore block on "smithi104-lv-2-data-08rssq" 2022-04-02T03:15:10.365 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:15:10.369325 I | op-osd: OSD orchestration status for node smithi104-lv-3-data-0jlrxt is "orchestrating" 2022-04-02T03:15:10.736 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:15:10.739822 I | op-osd: OSD orchestration status for PVC smithi104-lv-3-data-0jlrxt is "orchestrating" 2022-04-02T03:15:11.609 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph osd stat -f json 2022-04-02T03:15:11.832 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:15:11.835756 I | op-osd: OSD orchestration status for PVC smithi142-lv-3-data-05pxj5 is "completed" 2022-04-02T03:15:11.832 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:15:11.835783 I | op-osd: creating OSD 4 on PVC "smithi142-lv-3-data-05pxj5" 2022-04-02T03:15:11.832 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:15:11.835798 I | op-osd: OSD will have its main bluestore block on "smithi142-lv-3-data-05pxj5" 2022-04-02T03:15:12.032 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:15:12.035947 I | op-osd: OSD orchestration status for PVC smithi104-lv-4-data-0nmqwk is "completed" 2022-04-02T03:15:12.032 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:15:12.035989 I | op-osd: creating OSD 2 on PVC "smithi104-lv-4-data-0nmqwk" 2022-04-02T03:15:12.033 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:15:12.036020 I | op-osd: OSD will have its main bluestore block on "smithi104-lv-4-data-0nmqwk" 2022-04-02T03:15:12.335 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:15:12.338722 I | op-osd: OSD orchestration status for PVC smithi142-lv-2-data-0qdbgg is "completed" 2022-04-02T03:15:12.335 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:15:12.338742 I | op-osd: creating OSD 3 on PVC "smithi142-lv-2-data-0qdbgg" 2022-04-02T03:15:12.336 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:15:12.338750 I | op-osd: OSD will have its main bluestore block on "smithi142-lv-2-data-0qdbgg" 2022-04-02T03:15:12.450 INFO:teuthology.orchestra.run.smithi029.stdout: 2022-04-02T03:15:12.468 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":13,"num_osds":11,"num_up_osds":0,"osd_up_since":0,"num_in_osds":11,"osd_in_since":1648869310,"num_remapped_pgs":0} 2022-04-02T03:15:12.469 INFO:tasks.rook: have 0/12 OSDs 2022-04-02T03:15:12.680 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:15:12.683616 I | op-osd: OSD orchestration status for PVC smithi029-lv-1-data-0xzkjz is "completed" 2022-04-02T03:15:12.680 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:15:12.683645 I | op-osd: creating OSD 5 on PVC "smithi029-lv-1-data-0xzkjz" 2022-04-02T03:15:12.680 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:15:12.683658 I | op-osd: OSD will have its main bluestore block on "smithi029-lv-1-data-0xzkjz" 2022-04-02T03:15:14.362 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:15:14.366086 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi029": the object has been modified; please apply your changes to the latest version and try again 2022-04-02T03:15:14.964 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:15:14.967921 I | op-osd: OSD orchestration status for PVC smithi142-lv-1-data-08v2jt is "completed" 2022-04-02T03:15:14.965 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:15:14.967939 I | op-osd: creating OSD 6 on PVC "smithi142-lv-1-data-08v2jt" 2022-04-02T03:15:14.965 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:15:14.967953 I | op-osd: OSD will have its main bluestore block on "smithi142-lv-1-data-08v2jt" 2022-04-02T03:15:17.447 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:15:17.449351 I | op-osd: OSD orchestration status for PVC smithi029-lv-2-data-084vgd is "completed" 2022-04-02T03:15:17.447 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:15:17.449378 I | op-osd: creating OSD 7 on PVC "smithi029-lv-2-data-084vgd" 2022-04-02T03:15:17.447 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:15:17.449388 I | op-osd: OSD will have its main bluestore block on "smithi029-lv-2-data-084vgd" 2022-04-02T03:15:19.004 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:15:19.006588 I | op-osd: OSD orchestration status for PVC smithi142-lv-4-data-0hs4jx is "completed" 2022-04-02T03:15:19.004 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:15:19.006626 I | op-osd: creating OSD 8 on PVC "smithi142-lv-4-data-0hs4jx" 2022-04-02T03:15:19.005 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:15:19.006647 I | op-osd: OSD will have its main bluestore block on "smithi142-lv-4-data-0hs4jx" 2022-04-02T03:15:20.845 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:15:20.847395 I | op-osd: OSD orchestration status for PVC smithi029-lv-4-data-0sxfm8 is "completed" 2022-04-02T03:15:20.846 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:15:20.847432 I | op-osd: creating OSD 9 on PVC "smithi029-lv-4-data-0sxfm8" 2022-04-02T03:15:20.846 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:15:20.847450 I | op-osd: OSD will have its main bluestore block on "smithi029-lv-4-data-0sxfm8" 2022-04-02T03:15:22.470 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph osd stat -f json 2022-04-02T03:15:22.953 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:15:22.955459 I | op-osd: OSD orchestration status for PVC smithi029-lv-3-data-0f2zlj is "completed" 2022-04-02T03:15:22.954 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:15:22.955481 I | op-osd: creating OSD 10 on PVC "smithi029-lv-3-data-0f2zlj" 2022-04-02T03:15:22.954 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:15:22.955489 I | op-osd: OSD will have its main bluestore block on "smithi029-lv-3-data-0f2zlj" 2022-04-02T03:15:25.942 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:15:25.941021 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". failed to reconcile cluster "rook-ceph": failed to configure local ceph cluster: failed to create cluster: failed to start ceph osds: 1 failures encountered while running osds on nodes in namespace "rook-ceph". 2022-04-02T03:15:25.942 INFO:tasks.rook.operator.smithi029.stdout:failed to provision OSD(s) on PVC smithi104-lv-3-data-0jlrxt. &{OSDs:[] Status:failed PvcBackedOSD:true Message:failed to configure devices: failed to initialize devices on PVC: failed to run ceph-volume. Running command: /usr/bin/ceph-authtool --gen-print-key 2022-04-02T03:15:25.943 INFO:tasks.rook.operator.smithi029.stdout:Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new f2994011-a714-47a8-a17d-b0dbbffec787 2022-04-02T03:15:25.943 INFO:tasks.rook.operator.smithi029.stdout: stderr: unable to get monitor info from DNS SRV with service name: ceph-mon 2022-04-02T03:15:25.943 INFO:tasks.rook.operator.smithi029.stdout: stderr: 2022-04-02T03:15:05.071+0000 7fd488959700 -1 failed for service _ceph-mon._tcp 2022-04-02T03:15:25.943 INFO:tasks.rook.operator.smithi029.stdout: stderr: 2022-04-02T03:15:05.071+0000 7fd488959700 -1 monclient: get_monmap_and_config cannot identify monitors to contact 2022-04-02T03:15:25.944 INFO:tasks.rook.operator.smithi029.stdout: stderr: [errno 2] RADOS object not found (error connecting to the cluster) 2022-04-02T03:15:25.944 INFO:tasks.rook.operator.smithi029.stdout:Traceback (most recent call last): 2022-04-02T03:15:25.944 INFO:tasks.rook.operator.smithi029.stdout: File "/usr/sbin/ceph-volume", line 11, in 2022-04-02T03:15:25.944 INFO:tasks.rook.operator.smithi029.stdout: load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2022-04-02T03:15:25.945 INFO:tasks.rook.operator.smithi029.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2022-04-02T03:15:25.945 INFO:tasks.rook.operator.smithi029.stdout: self.main(self.argv) 2022-04-02T03:15:25.945 INFO:tasks.rook.operator.smithi029.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2022-04-02T03:15:25.945 INFO:tasks.rook.operator.smithi029.stdout: return f(*a, **kw) 2022-04-02T03:15:25.945 INFO:tasks.rook.operator.smithi029.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2022-04-02T03:15:25.946 INFO:tasks.rook.operator.smithi029.stdout: terminal.dispatch(self.mapper, subcommand_args) 2022-04-02T03:15:25.946 INFO:tasks.rook.operator.smithi029.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2022-04-02T03:15:25.946 INFO:tasks.rook.operator.smithi029.stdout: instance.main() 2022-04-02T03:15:25.946 INFO:tasks.rook.operator.smithi029.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/main.py", line 32, in main 2022-04-02T03:15:25.947 INFO:tasks.rook.operator.smithi029.stdout: terminal.dispatch(self.mapper, self.argv) 2022-04-02T03:15:25.947 INFO:tasks.rook.operator.smithi029.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2022-04-02T03:15:25.947 INFO:tasks.rook.operator.smithi029.stdout: instance.main() 2022-04-02T03:15:25.947 INFO:tasks.rook.operator.smithi029.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 169, in main 2022-04-02T03:15:25.954 INFO:tasks.rook.operator.smithi029.stdout: self.safe_prepare(self.args) 2022-04-02T03:15:25.955 INFO:tasks.rook.operator.smithi029.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 91, in safe_prepare 2022-04-02T03:15:25.955 INFO:tasks.rook.operator.smithi029.stdout: self.prepare() 2022-04-02T03:15:25.955 INFO:tasks.rook.operator.smithi029.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2022-04-02T03:15:25.955 INFO:tasks.rook.operator.smithi029.stdout: return func(*a, **kw) 2022-04-02T03:15:25.956 INFO:tasks.rook.operator.smithi029.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 125, in prepare 2022-04-02T03:15:25.956 INFO:tasks.rook.operator.smithi029.stdout: osd_fsid, json.dumps(secrets)) 2022-04-02T03:15:25.956 INFO:tasks.rook.operator.smithi029.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/util/prepare.py", line 173, in create_id 2022-04-02T03:15:25.956 INFO:tasks.rook.operator.smithi029.stdout: raise RuntimeError('Unable to create a new OSD id') 2022-04-02T03:15:25.957 INFO:tasks.rook.operator.smithi029.stdout:RuntimeError: Unable to create a new OSD id. debug logs below: 2022-04-02T03:15:25.957 INFO:tasks.rook.operator.smithi029.stdout:[2022-04-02 03:15:04,570][ceph_volume.main][INFO ] Running command: ceph-volume --log-path /var/log/ceph/smithi104-lv-3-data-0jlrxt raw prepare --bluestore --data /dev/mapper/vg_nvme-lv_3 2022-04-02T03:15:25.957 INFO:tasks.rook.operator.smithi029.stdout:[2022-04-02 03:15:04,571][ceph_volume.process][INFO ] Running command: /usr/bin/lsblk -plno KNAME,NAME,TYPE 2022-04-02T03:15:25.957 INFO:tasks.rook.operator.smithi029.stdout:[2022-04-02 03:15:04,581][ceph_volume.process][INFO ] stdout /dev/loop0 /dev/loop0 loop 2022-04-02T03:15:25.957 INFO:tasks.rook.operator.smithi029.stdout:[2022-04-02 03:15:04,582][ceph_volume.process][INFO ] stdout /dev/loop1 /dev/loop1 loop 2022-04-02T03:15:25.958 INFO:tasks.rook.operator.smithi029.stdout:[2022-04-02 03:15:04,582][ceph_volume.process][INFO ] stdout /dev/loop2 /dev/loop2 loop 2022-04-02T03:15:25.958 INFO:tasks.rook.operator.smithi029.stdout:[2022-04-02 03:15:04,582][ceph_volume.process][INFO ] stdout /dev/loop3 /dev/loop3 loop 2022-04-02T03:15:25.958 INFO:tasks.rook.operator.smithi029.stdout:[2022-04-02 03:15:04,582][ceph_volume.process][INFO ] stdout /dev/sda /dev/sda disk 2022-04-02T03:15:25.958 INFO:tasks.rook.operator.smithi029.stdout:[2022-04-02 03:15:04,582][ceph_volume.process][INFO ] stdout /dev/sda1 /dev/sda1 part 2022-04-02T03:15:25.958 INFO:tasks.rook.operator.smithi029.stdout:[2022-04-02 03:15:04,582][ceph_volume.process][INFO ] stdout /dev/dm-0 /dev/mapper/vg_nvme-lv_1 lvm 2022-04-02T03:15:25.959 INFO:tasks.rook.operator.smithi029.stdout:[2022-04-02 03:15:04,582][ceph_volume.process][INFO ] stdout /dev/dm-1 /dev/mapper/vg_nvme-lv_2 lvm 2022-04-02T03:15:25.959 INFO:tasks.rook.operator.smithi029.stdout:[2022-04-02 03:15:04,582][ceph_volume.process][INFO ] stdout /dev/dm-2 /dev/mapper/vg_nvme-lv_3 lvm 2022-04-02T03:15:25.959 INFO:tasks.rook.operator.smithi029.stdout:[2022-04-02 03:15:04,582][ceph_volume.process][INFO ] stdout /dev/dm-3 /dev/mapper/vg_nvme-lv_4 lvm 2022-04-02T03:15:25.959 INFO:tasks.rook.operator.smithi029.stdout:[2022-04-02 03:15:04,582][ceph_volume.process][INFO ] stdout /dev/dm-4 /dev/mapper/vg_nvme-lv_5 lvm 2022-04-02T03:15:25.959 INFO:tasks.rook.operator.smithi029.stdout:[2022-04-02 03:15:04,582][ceph_volume.process][INFO ] stdout /dev/nvme0n1 /dev/nvme0n1 disk 2022-04-02T03:15:25.960 INFO:tasks.rook.operator.smithi029.stdout:[2022-04-02 03:15:04,582][ceph_volume.process][INFO ] stdout /dev/nvme1n1 /dev/nvme1n1 disk 2022-04-02T03:15:25.960 INFO:tasks.rook.operator.smithi029.stdout:[2022-04-02 03:15:04,582][ceph_volume.process][INFO ] stdout /dev/nvme2n1 /dev/nvme2n1 disk 2022-04-02T03:15:25.960 INFO:tasks.rook.operator.smithi029.stdout:[2022-04-02 03:15:04,582][ceph_volume.process][INFO ] stdout /dev/nvme3n1 /dev/nvme3n1 disk 2022-04-02T03:15:25.960 INFO:tasks.rook.operator.smithi029.stdout:[2022-04-02 03:15:04,583][ceph_volume.process][INFO ] stdout /dev/nvme4n1 /dev/nvme4n1 disk 2022-04-02T03:15:25.960 INFO:tasks.rook.operator.smithi029.stdout:[2022-04-02 03:15:04,596][ceph_volume.util.system][WARNING] Executable lvs not found on the host, will return lvs as-is 2022-04-02T03:15:25.961 INFO:tasks.rook.operator.smithi029.stdout:[2022-04-02 03:15:04,596][ceph_volume.process][INFO ] Running command: lvs --noheadings --readonly --separator=";" -a --units=b --nosuffix -S lv_path=/dev/mapper/vg_nvme-lv_3 -o lv_tags,lv_path,lv_name,vg_name,lv_uuid,lv_size 2022-04-02T03:15:25.961 INFO:tasks.rook.operator.smithi029.stdout:[2022-04-02 03:15:04,910][ceph_volume.process][INFO ] Running command: /usr/bin/lsblk --nodeps -P -o NAME,KNAME,MAJ:MIN,FSTYPE,MOUNTPOINT,LABEL,UUID,RO,RM,MODEL,SIZE,STATE,OWNER,GROUP,MODE,ALIGNMENT,PHY-SEC,LOG-SEC,ROTA,SCHED,TYPE,DISC-ALN,DISC-GRAN,DISC-MAX,DISC-ZERO,PKNAME,PARTLABEL /dev/mapper/vg_nvme-lv_3 2022-04-02T03:15:25.961 INFO:tasks.rook.operator.smithi029.stdout:[2022-04-02 03:15:04,915][ceph_volume.process][INFO ] stdout NAME="vg_nvme-lv_3" KNAME="dm-2" MAJ:MIN="253:2" FSTYPE="" MOUNTPOINT="" LABEL="" UUID="" RO="0" RM="0" MODEL="" SIZE="89.4G" STATE="running" OWNER="root" GROUP="disk" MODE="brw-rw----" ALIGNMENT="0" PHY-SEC="512" LOG-SEC="512" ROTA="0" SCHED="" TYPE="lvm" DISC-ALN="0" DISC-GRAN="512B" DISC-MAX="2T" DISC-ZERO="0" PKNAME="" PARTLABEL="" 2022-04-02T03:15:25.961 INFO:tasks.rook.operator.smithi029.stdout:[2022-04-02 03:15:04,916][ceph_volume.process][INFO ] Running command: /usr/sbin/blkid -c /dev/null -p /dev/mapper/vg_nvme-lv_3 2022-04-02T03:15:25.961 INFO:tasks.rook.operator.smithi029.stdout:[2022-04-02 03:15:04,919][ceph_volume.util.disk][INFO ] opening device /dev/mapper/vg_nvme-lv_3 to check for BlueStore label 2022-04-02T03:15:25.962 INFO:tasks.rook.operator.smithi029.stdout:[2022-04-02 03:15:04,919][ceph_volume.util.disk][INFO ] opening device /dev/mapper/vg_nvme-lv_3 to check for BlueStore label 2022-04-02T03:15:25.962 INFO:tasks.rook.operator.smithi029.stdout:[2022-04-02 03:15:04,920][ceph_volume.process][INFO ] Running command: /usr/sbin/udevadm info --query=property /dev/mapper/vg_nvme-lv_3 2022-04-02T03:15:25.962 INFO:tasks.rook.operator.smithi029.stdout:[2022-04-02 03:15:04,926][ceph_volume.process][INFO ] stdout DEVLINKS=/dev/mapper/vg_nvme-lv_3 /dev/vg_nvme/lv_3 /dev/disk/by-id/dm-name-vg_nvme-lv_3 /dev/disk/by-id/dm-uuid-LVM-nNFcexrtq2jlNGKS14RaFdSTNQtJ1XS0lpWXInTLFSJNsgbq4ZZaUEAD8n2StRU9 2022-04-02T03:15:25.962 INFO:tasks.rook.operator.smithi029.stdout:[2022-04-02 03:15:04,927][ceph_volume.process][INFO ] stdout DEVNAME=/dev/dm-2 2022-04-02T03:15:25.963 INFO:tasks.rook.operator.smithi029.stdout:[2022-04-02 03:15:04,927][ceph_volume.process][INFO ] stdout DEVPATH=/devices/virtual/block/dm-2 2022-04-02T03:15:25.963 INFO:tasks.rook.operator.smithi029.stdout:[2022-04-02 03:15:04,927][ceph_volume.process][INFO ] stdout DEVTYPE=disk 2022-04-02T03:15:25.963 INFO:tasks.rook.operator.smithi029.stdout:[2022-04-02 03:15:04,927][ceph_volume.process][INFO ] stdout DM_LV_NAME=lv_3 2022-04-02T03:15:25.963 INFO:tasks.rook.operator.smithi029.stdout:[2022-04-02 03:15:04,927][ceph_volume.process][INFO ] stdout DM_NAME=vg_nvme-lv_3 2022-04-02T03:15:25.964 INFO:tasks.rook.operator.smithi029.stdout:[2022-04-02 03:15:04,927][ceph_volume.process][INFO ] stdout DM_SUSPENDED=0 2022-04-02T03:15:25.964 INFO:tasks.rook.operator.smithi029.stdout:[2022-04-02 03:15:04,927][ceph_volume.process][INFO ] stdout DM_UDEV_DISABLE_LIBRARY_FALLBACK_FLAG=1 2022-04-02T03:15:25.964 INFO:tasks.rook.operator.smithi029.stdout:[2022-04-02 03:15:04,927][ceph_volume.process][INFO ] stdout DM_UDEV_PRIMARY_SOURCE_FLAG=1 2022-04-02T03:15:25.964 INFO:tasks.rook.operator.smithi029.stdout:[2022-04-02 03:15:04,927][ceph_volume.process][INFO ] stdout DM_UDEV_RULES=1 2022-04-02T03:15:25.964 INFO:tasks.rook.operator.smithi029.stdout:[2022-04-02 03:15:04,927][ceph_volume.process][INFO ] stdout DM_UDEV_RULES_VSN=2 2022-04-02T03:15:25.965 INFO:tasks.rook.operator.smithi029.stdout:[2022-04-02 03:15:04,927][ceph_volume.process][INFO ] stdout DM_UUID=LVM-nNFcexrtq2jlNGKS14RaFdSTNQtJ1XS0lpWXInTLFSJNsgbq4ZZaUEAD8n2StRU9 2022-04-02T03:15:25.965 INFO:tasks.rook.operator.smithi029.stdout:[2022-04-02 03:15:04,927][ceph_volume.process][INFO ] stdout DM_VG_NAME=vg_nvme 2022-04-02T03:15:25.965 INFO:tasks.rook.operator.smithi029.stdout:[2022-04-02 03:15:04,927][ceph_volume.process][INFO ] stdout MAJOR=253 2022-04-02T03:15:25.965 INFO:tasks.rook.operator.smithi029.stdout:[2022-04-02 03:15:04,927][ceph_volume.process][INFO ] stdout MINOR=2 2022-04-02T03:15:25.966 INFO:tasks.rook.operator.smithi029.stdout:[2022-04-02 03:15:04,927][ceph_volume.process][INFO ] stdout SUBSYSTEM=block 2022-04-02T03:15:25.966 INFO:tasks.rook.operator.smithi029.stdout:[2022-04-02 03:15:04,927][ceph_volume.process][INFO ] stdout TAGS=:systemd: 2022-04-02T03:15:25.966 INFO:tasks.rook.operator.smithi029.stdout:[2022-04-02 03:15:04,927][ceph_volume.process][INFO ] stdout USEC_INITIALIZED=227059161 2022-04-02T03:15:25.966 INFO:tasks.rook.operator.smithi029.stdout:[2022-04-02 03:15:04,928][ceph_volume.process][INFO ] Running command: /usr/bin/ceph-authtool --gen-print-key 2022-04-02T03:15:25.967 INFO:tasks.rook.operator.smithi029.stdout:[2022-04-02 03:15:04,944][ceph_volume.process][INFO ] stdout AQC4v0diJP88OBAARkHyrc1ioE7ljbCaDa1trQ== 2022-04-02T03:15:25.967 INFO:tasks.rook.operator.smithi029.stdout:[2022-04-02 03:15:04,945][ceph_volume.process][INFO ] Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new f2994011-a714-47a8-a17d-b0dbbffec787 2022-04-02T03:15:25.967 INFO:tasks.rook.operator.smithi029.stdout:[2022-04-02 03:15:05,077][ceph_volume.process][INFO ] stderr unable to get monitor info from DNS SRV with service name: ceph-mon 2022-04-02T03:15:25.967 INFO:tasks.rook.operator.smithi029.stdout:[2022-04-02 03:15:05,078][ceph_volume.process][INFO ] stderr 2022-04-02T03:15:05.071+0000 7fd488959700 -1 failed for service _ceph-mon._tcp 2022-04-02T03:15:25.967 INFO:tasks.rook.operator.smithi029.stdout:[2022-04-02 03:15:05,078][ceph_volume.process][INFO ] stderr 2022-04-02T03:15:05.071+0000 7fd488959700 -1 monclient: get_monmap_and_config cannot identify monitors to contact 2022-04-02T03:15:25.968 INFO:tasks.rook.operator.smithi029.stdout:[2022-04-02 03:15:05,078][ceph_volume.process][INFO ] stderr [errno 2] RADOS object not found (error connecting to the cluster) 2022-04-02T03:15:25.968 INFO:tasks.rook.operator.smithi029.stdout:[2022-04-02 03:15:05,078][ceph_volume.devices.raw.prepare][ERROR ] raw prepare was unable to complete 2022-04-02T03:15:25.968 INFO:tasks.rook.operator.smithi029.stdout:Traceback (most recent call last): 2022-04-02T03:15:25.968 INFO:tasks.rook.operator.smithi029.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 91, in safe_prepare 2022-04-02T03:15:25.968 INFO:tasks.rook.operator.smithi029.stdout: self.prepare() 2022-04-02T03:15:25.969 INFO:tasks.rook.operator.smithi029.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2022-04-02T03:15:25.969 INFO:tasks.rook.operator.smithi029.stdout: return func(*a, **kw) 2022-04-02T03:15:25.969 INFO:tasks.rook.operator.smithi029.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 125, in prepare 2022-04-02T03:15:25.969 INFO:tasks.rook.operator.smithi029.stdout: osd_fsid, json.dumps(secrets)) 2022-04-02T03:15:25.969 INFO:tasks.rook.operator.smithi029.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/util/prepare.py", line 173, in create_id 2022-04-02T03:15:25.970 INFO:tasks.rook.operator.smithi029.stdout: raise RuntimeError('Unable to create a new OSD id') 2022-04-02T03:15:25.970 INFO:tasks.rook.operator.smithi029.stdout:RuntimeError: Unable to create a new OSD id 2022-04-02T03:15:25.970 INFO:tasks.rook.operator.smithi029.stdout:[2022-04-02 03:15:05,078][ceph_volume.devices.raw.prepare][INFO ] will rollback OSD ID creation 2022-04-02T03:15:25.970 INFO:tasks.rook.operator.smithi029.stdout:[2022-04-02 03:15:05,079][ceph_volume][ERROR ] exception caught by decorator 2022-04-02T03:15:25.970 INFO:tasks.rook.operator.smithi029.stdout:Traceback (most recent call last): 2022-04-02T03:15:25.971 INFO:tasks.rook.operator.smithi029.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2022-04-02T03:15:25.971 INFO:tasks.rook.operator.smithi029.stdout: return f(*a, **kw) 2022-04-02T03:15:25.971 INFO:tasks.rook.operator.smithi029.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2022-04-02T03:15:25.971 INFO:tasks.rook.operator.smithi029.stdout: terminal.dispatch(self.mapper, subcommand_args) 2022-04-02T03:15:25.971 INFO:tasks.rook.operator.smithi029.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2022-04-02T03:15:25.972 INFO:tasks.rook.operator.smithi029.stdout: instance.main() 2022-04-02T03:15:25.972 INFO:tasks.rook.operator.smithi029.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/main.py", line 32, in main 2022-04-02T03:15:25.972 INFO:tasks.rook.operator.smithi029.stdout: terminal.dispatch(self.mapper, self.argv) 2022-04-02T03:15:25.972 INFO:tasks.rook.operator.smithi029.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2022-04-02T03:15:25.972 INFO:tasks.rook.operator.smithi029.stdout: instance.main() 2022-04-02T03:15:25.973 INFO:tasks.rook.operator.smithi029.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 169, in main 2022-04-02T03:15:25.973 INFO:tasks.rook.operator.smithi029.stdout: self.safe_prepare(self.args) 2022-04-02T03:15:25.973 INFO:tasks.rook.operator.smithi029.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 91, in safe_prepare 2022-04-02T03:15:25.973 INFO:tasks.rook.operator.smithi029.stdout: self.prepare() 2022-04-02T03:15:25.974 INFO:tasks.rook.operator.smithi029.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2022-04-02T03:15:25.974 INFO:tasks.rook.operator.smithi029.stdout: return func(*a, **kw) 2022-04-02T03:15:25.974 INFO:tasks.rook.operator.smithi029.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 125, in prepare 2022-04-02T03:15:25.974 INFO:tasks.rook.operator.smithi029.stdout: osd_fsid, json.dumps(secrets)) 2022-04-02T03:15:25.974 INFO:tasks.rook.operator.smithi029.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/util/prepare.py", line 173, in create_id 2022-04-02T03:15:25.975 INFO:tasks.rook.operator.smithi029.stdout: raise RuntimeError('Unable to create a new OSD id') 2022-04-02T03:15:25.975 INFO:tasks.rook.operator.smithi029.stdout:RuntimeError: Unable to create a new OSD id 2022-04-02T03:15:25.975 INFO:tasks.rook.operator.smithi029.stdout:: exit status 1} 2022-04-02T03:15:25.975 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:15:25.946445 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-04-02T03:15:26.123 INFO:teuthology.orchestra.run.smithi029.stdout: 2022-04-02T03:15:26.146 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":14,"num_osds":12,"num_up_osds":0,"osd_up_since":0,"num_in_osds":12,"osd_in_since":1648869312,"num_remapped_pgs":0} 2022-04-02T03:15:26.147 INFO:tasks.rook: have 0/12 OSDs 2022-04-02T03:15:26.889 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:15:26.891462 I | op-mon: parsing mon endpoints: a=10.102.133.199:6789,b=10.107.147.59:6789,c=10.96.208.36:6789 2022-04-02T03:15:27.382 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:15:27.384489 I | ceph-spec: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a... 2022-04-02T03:15:36.148 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph osd stat -f json 2022-04-02T03:15:38.594 INFO:teuthology.orchestra.run.smithi029.stdout: 2022-04-02T03:15:38.621 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":17,"num_osds":12,"num_up_osds":1,"osd_up_since":1648869337,"num_in_osds":12,"osd_in_since":1648869312,"num_remapped_pgs":0} 2022-04-02T03:15:38.621 INFO:tasks.rook: have 1/12 OSDs 2022-04-02T03:15:48.623 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph osd stat -f json 2022-04-02T03:15:49.445 INFO:teuthology.orchestra.run.smithi029.stdout: 2022-04-02T03:15:49.464 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":23,"num_osds":12,"num_up_osds":7,"osd_up_since":1648869348,"num_in_osds":12,"osd_in_since":1648869312,"num_remapped_pgs":0} 2022-04-02T03:15:49.465 INFO:tasks.rook: have 7/12 OSDs 2022-04-02T03:15:50.128 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:15:50.128342 I | ceph-spec: detected ceph image version: "17.0.0-11449 quincy" 2022-04-02T03:15:50.129 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:15:50.128368 I | ceph-cluster-controller: validating ceph version from provided image 2022-04-02T03:15:50.494 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:15:50.493980 I | op-mon: parsing mon endpoints: a=10.102.133.199:6789,b=10.107.147.59:6789,c=10.96.208.36:6789 2022-04-02T03:15:50.786 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:15:50.786015 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-04-02T03:15:50.786 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:15:50.786301 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-04-02T03:15:51.252 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:15:51.252727 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-11449 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a" 2022-04-02T03:15:52.059 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:15:52.059706 I | op-mon: start running mons 2022-04-02T03:15:52.433 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:15:52.432952 I | op-mon: parsing mon endpoints: a=10.102.133.199:6789,b=10.107.147.59:6789,c=10.96.208.36:6789 2022-04-02T03:15:54.831 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:15:54.830642 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.102.133.199:6789","10.107.147.59:6789","10.96.208.36:6789"]}] data:a=10.102.133.199:6789,b=10.107.147.59:6789,c=10.96.208.36:6789 mapping:{"node":{"a":{"Name":"smithi104","Hostname":"smithi104","Address":"172.21.15.104"},"b":{"Name":"smithi142","Hostname":"smithi142","Address":"172.21.15.142"},"c":{"Name":"smithi029","Hostname":"smithi029","Address":"172.21.15.29"}}} maxMonId:2] 2022-04-02T03:15:55.650 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:15:55.649512 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-04-02T03:15:55.650 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:15:55.649860 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-04-02T03:15:56.254 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:15:56.253662 I | op-mon: targeting the mon count 3 2022-04-02T03:15:56.260 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:15:56.260329 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-04-02T03:15:56.677 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:15:56.676637 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-04-02T03:15:56.677 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:15:56.676656 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-04-02T03:15:57.180 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:15:57.179345 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-04-02T03:15:57.180 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:15:57.179390 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-04-02T03:15:57.526 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:15:57.526090 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-04-02T03:15:57.526 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:15:57.526128 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-04-02T03:15:59.466 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph osd stat -f json 2022-04-02T03:15:59.611 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:15:59.610244 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-04-02T03:15:59.612 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:15:59.611284 I | op-config: deleting "log file" option from the mon configuration database 2022-04-02T03:16:00.097 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:16:00.096717 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-04-02T03:16:00.098 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:16:00.096752 I | op-mon: checking for basic quorum with existing mons 2022-04-02T03:16:01.030 INFO:teuthology.orchestra.run.smithi029.stdout: 2022-04-02T03:16:01.049 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":24,"num_osds":12,"num_up_osds":7,"osd_up_since":1648869348,"num_in_osds":12,"osd_in_since":1648869312,"num_remapped_pgs":0} 2022-04-02T03:16:01.049 INFO:tasks.rook: have 7/12 OSDs 2022-04-02T03:16:01.632 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:16:01.631663 I | op-mon: mon "a" ip is 10.102.133.199 2022-04-02T03:16:02.899 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:16:02.898765 I | op-mon: mon "b" ip is 10.107.147.59 2022-04-02T03:16:03.324 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:16:03.324112 I | op-mon: mon "c" ip is 10.96.208.36 2022-04-02T03:16:03.758 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:16:03.756738 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.102.133.199:6789","10.107.147.59:6789","10.96.208.36:6789"]}] data:a=10.102.133.199:6789,b=10.107.147.59:6789,c=10.96.208.36:6789 mapping:{"node":{"a":{"Name":"smithi104","Hostname":"smithi104","Address":"172.21.15.104"},"b":{"Name":"smithi142","Hostname":"smithi142","Address":"172.21.15.142"},"c":{"Name":"smithi029","Hostname":"smithi029","Address":"172.21.15.29"}}} maxMonId:2] 2022-04-02T03:16:04.470 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:16:04.469181 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-04-02T03:16:04.470 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:16:04.469481 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-04-02T03:16:04.701 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:16:04.700817 I | op-mon: deployment for mon rook-ceph-mon-a already exists. updating if needed 2022-04-02T03:16:04.710 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:16:04.709699 I | op-k8sutil: deployment "rook-ceph-mon-a" did not change, nothing to update 2022-04-02T03:16:04.711 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:16:04.709720 I | op-mon: waiting for mon quorum with [a b c] 2022-04-02T03:16:04.739 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:16:04.738224 I | op-mon: mons running: [a b c] 2022-04-02T03:16:05.148 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:16:05.147739 I | op-mon: Monitors in quorum: [a b c] 2022-04-02T03:16:05.156 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:16:05.156374 I | op-mon: deployment for mon rook-ceph-mon-b already exists. updating if needed 2022-04-02T03:16:05.168 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:16:05.168232 I | op-k8sutil: deployment "rook-ceph-mon-b" did not change, nothing to update 2022-04-02T03:16:05.169 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:16:05.168263 I | op-mon: waiting for mon quorum with [a b c] 2022-04-02T03:16:05.203 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:16:05.202643 I | op-mon: mons running: [a b c] 2022-04-02T03:16:05.605 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:16:05.604889 I | op-mon: Monitors in quorum: [a b c] 2022-04-02T03:16:05.614 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:16:05.613551 I | op-mon: deployment for mon rook-ceph-mon-c already exists. updating if needed 2022-04-02T03:16:05.625 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:16:05.624134 I | op-k8sutil: deployment "rook-ceph-mon-c" did not change, nothing to update 2022-04-02T03:16:05.625 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:16:05.624159 I | op-mon: waiting for mon quorum with [a b c] 2022-04-02T03:16:05.657 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:16:05.657281 I | op-mon: mons running: [a b c] 2022-04-02T03:16:06.058 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:16:06.057326 I | op-mon: Monitors in quorum: [a b c] 2022-04-02T03:16:06.058 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:16:06.057357 I | op-mon: mons created: 3 2022-04-02T03:16:06.481 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:16:06.480367 I | op-mon: waiting for mon quorum with [a b c] 2022-04-02T03:16:07.334 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:16:07.333191 I | op-mon: mons running: [a b c] 2022-04-02T03:16:07.733 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:16:07.732816 I | op-mon: Monitors in quorum: [a b c] 2022-04-02T03:16:07.734 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:16:07.732846 I | cephclient: getting or creating ceph auth key "client.csi-rbd-provisioner" 2022-04-02T03:16:08.260 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:16:08.258804 I | cephclient: getting or creating ceph auth key "client.csi-rbd-node" 2022-04-02T03:16:08.717 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:16:08.716177 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-provisioner" 2022-04-02T03:16:09.186 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:16:09.185328 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-node" 2022-04-02T03:16:09.663 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:16:09.663033 I | ceph-csi: created kubernetes csi secrets for cluster "rook-ceph" 2022-04-02T03:16:09.664 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:16:09.663053 I | cephclient: getting or creating ceph auth key "client.crash" 2022-04-02T03:16:10.124 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:16:10.123377 I | ceph-crashcollector-controller: created kubernetes crash collector secret for cluster "rook-ceph" 2022-04-02T03:16:10.125 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:16:10.123411 I | cephclient: create rbd-mirror bootstrap peer token "client.rbd-mirror-peer" 2022-04-02T03:16:10.125 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:16:10.123420 I | cephclient: getting or creating ceph auth key "client.rbd-mirror-peer" 2022-04-02T03:16:10.600 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:16:10.599644 I | cephclient: successfully created rbd-mirror bootstrap peer token for cluster "rook-ceph" 2022-04-02T03:16:10.632 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:16:10.631187 I | op-mgr: start running mgr 2022-04-02T03:16:10.632 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:16:10.631222 I | cephclient: getting or creating ceph auth key "mgr.a" 2022-04-02T03:16:11.051 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph osd stat -f json 2022-04-02T03:16:11.112 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:16:11.111365 I | op-mgr: deployment for mgr rook-ceph-mgr-a already exists. updating if needed 2022-04-02T03:16:11.120 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:16:11.119441 I | op-k8sutil: deployment "rook-ceph-mgr-a" did not change, nothing to update 2022-04-02T03:16:11.123 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:16:11.122942 I | op-mgr: setting services to point to mgr "a" 2022-04-02T03:16:11.538 INFO:teuthology.orchestra.run.smithi029.stdout: 2022-04-02T03:16:11.557 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":28,"num_osds":12,"num_up_osds":9,"osd_up_since":1648869370,"num_in_osds":12,"osd_in_since":1648869312,"num_remapped_pgs":1} 2022-04-02T03:16:11.557 INFO:tasks.rook: have 9/12 OSDs 2022-04-02T03:16:11.625 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:16:11.624875 I | op-mgr: no need to update service "rook-ceph-mgr" 2022-04-02T03:16:11.626 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:16:11.624960 I | op-mgr: successful modules: balancer 2022-04-02T03:16:11.637 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:16:11.637319 I | op-osd: start running osds in namespace "rook-ceph" 2022-04-02T03:16:11.638 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:16:11.637351 I | op-osd: wait timeout for healthy OSDs during upgrade or restart is "10m0s" 2022-04-02T03:16:11.737 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:16:11.736153 I | op-osd: start provisioning the OSDs on PVCs, if needed 2022-04-02T03:16:11.828 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:16:11.827404 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi029-lv-1" 2022-04-02T03:16:11.829 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:16:11.827452 I | op-osd: OSD PVC "smithi029-lv-1-data-0xzkjz" already exists 2022-04-02T03:16:11.829 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:16:11.827468 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi029-lv-2" 2022-04-02T03:16:11.829 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:16:11.827485 I | op-osd: OSD PVC "smithi029-lv-2-data-084vgd" already exists 2022-04-02T03:16:11.829 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:16:11.827497 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi029-lv-3" 2022-04-02T03:16:11.834 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:16:11.827509 I | op-osd: OSD PVC "smithi029-lv-3-data-0f2zlj" already exists 2022-04-02T03:16:11.834 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:16:11.827519 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi029-lv-4" 2022-04-02T03:16:11.835 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:16:11.827529 I | op-osd: OSD PVC "smithi029-lv-4-data-0sxfm8" already exists 2022-04-02T03:16:11.835 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:16:11.827536 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi104-lv-1" 2022-04-02T03:16:11.835 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:16:11.827548 I | op-osd: OSD PVC "smithi104-lv-1-data-0jfbpq" already exists 2022-04-02T03:16:11.836 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:16:11.827565 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi104-lv-2" 2022-04-02T03:16:11.836 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:16:11.827577 I | op-osd: OSD PVC "smithi104-lv-2-data-08rssq" already exists 2022-04-02T03:16:11.836 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:16:11.827584 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi104-lv-3" 2022-04-02T03:16:11.837 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:16:11.827595 I | op-osd: OSD PVC "smithi104-lv-3-data-0jlrxt" already exists 2022-04-02T03:16:11.837 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:16:11.827602 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi104-lv-4" 2022-04-02T03:16:11.838 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:16:11.827614 I | op-osd: OSD PVC "smithi104-lv-4-data-0nmqwk" already exists 2022-04-02T03:16:11.839 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:16:11.827621 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi142-lv-1" 2022-04-02T03:16:11.839 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:16:11.827631 I | op-osd: OSD PVC "smithi142-lv-1-data-08v2jt" already exists 2022-04-02T03:16:11.839 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:16:11.827642 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi142-lv-2" 2022-04-02T03:16:11.839 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:16:11.827654 I | op-osd: OSD PVC "smithi142-lv-2-data-0qdbgg" already exists 2022-04-02T03:16:11.840 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:16:11.827663 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi142-lv-3" 2022-04-02T03:16:11.840 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:16:11.827673 I | op-osd: OSD PVC "smithi142-lv-3-data-05pxj5" already exists 2022-04-02T03:16:11.840 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:16:11.827681 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi142-lv-4" 2022-04-02T03:16:11.840 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:16:11.827692 I | op-osd: OSD PVC "smithi142-lv-4-data-0hs4jx" already exists 2022-04-02T03:16:11.863 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:16:11.861316 I | op-osd: OSD will have its main bluestore block on "smithi029-lv-1-data-0xzkjz" 2022-04-02T03:16:11.863 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:16:11.861339 I | op-osd: skipping OSD prepare job creation for PVC "smithi029-lv-1-data-0xzkjz" because OSD daemon using the PVC already exists 2022-04-02T03:16:11.864 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:16:11.861343 I | op-osd: OSD will have its main bluestore block on "smithi029-lv-2-data-084vgd" 2022-04-02T03:16:11.864 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:16:11.861345 I | op-osd: skipping OSD prepare job creation for PVC "smithi029-lv-2-data-084vgd" because OSD daemon using the PVC already exists 2022-04-02T03:16:11.864 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:16:11.861348 I | op-osd: OSD will have its main bluestore block on "smithi029-lv-3-data-0f2zlj" 2022-04-02T03:16:11.864 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:16:11.861351 I | op-osd: skipping OSD prepare job creation for PVC "smithi029-lv-3-data-0f2zlj" because OSD daemon using the PVC already exists 2022-04-02T03:16:11.864 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:16:11.861353 I | op-osd: OSD will have its main bluestore block on "smithi029-lv-4-data-0sxfm8" 2022-04-02T03:16:11.865 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:16:11.861356 I | op-osd: skipping OSD prepare job creation for PVC "smithi029-lv-4-data-0sxfm8" because OSD daemon using the PVC already exists 2022-04-02T03:16:11.865 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:16:11.861358 I | op-osd: OSD will have its main bluestore block on "smithi104-lv-1-data-0jfbpq" 2022-04-02T03:16:11.865 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:16:11.861361 I | op-osd: skipping OSD prepare job creation for PVC "smithi104-lv-1-data-0jfbpq" because OSD daemon using the PVC already exists 2022-04-02T03:16:11.865 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:16:11.861363 I | op-osd: OSD will have its main bluestore block on "smithi104-lv-2-data-08rssq" 2022-04-02T03:16:11.866 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:16:11.861366 I | op-osd: skipping OSD prepare job creation for PVC "smithi104-lv-2-data-08rssq" because OSD daemon using the PVC already exists 2022-04-02T03:16:11.866 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:16:11.861368 I | op-osd: OSD will have its main bluestore block on "smithi104-lv-3-data-0jlrxt" 2022-04-02T03:16:12.242 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:16:12.240898 I | op-k8sutil: Removing previous job rook-ceph-osd-prepare-smithi104-lv-3-data-0jlrxt to start a new one 2022-04-02T03:16:12.269 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:16:12.269089 I | op-k8sutil: batch job rook-ceph-osd-prepare-smithi104-lv-3-data-0jlrxt still exists 2022-04-02T03:16:13.228 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:16:13.227182 I | op-mgr: successful modules: mgr module(s) from the spec 2022-04-02T03:16:13.263 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:16:13.261905 I | op-mgr: successful modules: prometheus 2022-04-02T03:16:13.267 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:16:13.263102 I | op-mgr: successful modules: dashboard 2022-04-02T03:16:14.945 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:16:14.944165 I | op-mgr: successful modules: orchestrator modules 2022-04-02T03:16:15.276 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:16:15.275344 I | op-k8sutil: batch job rook-ceph-osd-prepare-smithi104-lv-3-data-0jlrxt deleted 2022-04-02T03:16:15.286 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:16:15.283399 I | op-osd: started OSD provisioning job for PVC "smithi104-lv-3-data-0jlrxt" 2022-04-02T03:16:15.286 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:16:15.283429 I | op-osd: OSD will have its main bluestore block on "smithi104-lv-4-data-0nmqwk" 2022-04-02T03:16:15.286 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:16:15.283438 I | op-osd: skipping OSD prepare job creation for PVC "smithi104-lv-4-data-0nmqwk" because OSD daemon using the PVC already exists 2022-04-02T03:16:15.287 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:16:15.283447 I | op-osd: OSD will have its main bluestore block on "smithi142-lv-1-data-08v2jt" 2022-04-02T03:16:15.287 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:16:15.283453 I | op-osd: skipping OSD prepare job creation for PVC "smithi142-lv-1-data-08v2jt" because OSD daemon using the PVC already exists 2022-04-02T03:16:15.287 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:16:15.283460 I | op-osd: OSD will have its main bluestore block on "smithi142-lv-2-data-0qdbgg" 2022-04-02T03:16:15.287 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:16:15.283466 I | op-osd: skipping OSD prepare job creation for PVC "smithi142-lv-2-data-0qdbgg" because OSD daemon using the PVC already exists 2022-04-02T03:16:15.288 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:16:15.283473 I | op-osd: OSD will have its main bluestore block on "smithi142-lv-3-data-05pxj5" 2022-04-02T03:16:15.288 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:16:15.283480 I | op-osd: skipping OSD prepare job creation for PVC "smithi142-lv-3-data-05pxj5" because OSD daemon using the PVC already exists 2022-04-02T03:16:15.288 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:16:15.283487 I | op-osd: OSD will have its main bluestore block on "smithi142-lv-4-data-0hs4jx" 2022-04-02T03:16:15.288 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:16:15.283495 I | op-osd: skipping OSD prepare job creation for PVC "smithi142-lv-4-data-0hs4jx" because OSD daemon using the PVC already exists 2022-04-02T03:16:15.288 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:16:15.283502 I | op-osd: start provisioning the OSDs on nodes, if needed 2022-04-02T03:16:15.289 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:16:15.283508 I | op-osd: no nodes are defined for configuring OSDs on raw devices 2022-04-02T03:16:15.311 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:16:15.310785 I | op-osd: OSD orchestration status for PVC smithi104-lv-3-data-0jlrxt is "starting" 2022-04-02T03:16:15.413 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:16:15.412166 I | op-osd: skipping osd checks for ok-to-stop 2022-04-02T03:16:15.456 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:16:15.455563 I | op-osd: updating OSD 0 on PVC "smithi104-lv-1-data-0jfbpq" 2022-04-02T03:16:15.457 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:16:15.455604 I | op-osd: OSD will have its main bluestore block on "smithi104-lv-1-data-0jfbpq" 2022-04-02T03:16:15.659 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:16:15.658610 I | op-osd: skipping osd checks for ok-to-stop 2022-04-02T03:16:15.664 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:16:15.663422 I | op-osd: updating OSD 1 on PVC "smithi104-lv-2-data-08rssq" 2022-04-02T03:16:15.664 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:16:15.663452 I | op-osd: OSD will have its main bluestore block on "smithi104-lv-2-data-08rssq" 2022-04-02T03:16:15.884 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:16:15.883206 I | op-osd: skipping osd checks for ok-to-stop 2022-04-02T03:16:15.892 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:16:15.890214 I | op-osd: updating OSD 10 on PVC "smithi029-lv-3-data-0f2zlj" 2022-04-02T03:16:15.893 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:16:15.890254 I | op-osd: OSD will have its main bluestore block on "smithi029-lv-3-data-0f2zlj" 2022-04-02T03:16:16.080 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:16:16.079425 I | op-osd: skipping osd checks for ok-to-stop 2022-04-02T03:16:16.086 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:16:16.085242 I | op-osd: updating OSD 2 on PVC "smithi104-lv-4-data-0nmqwk" 2022-04-02T03:16:16.086 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:16:16.085263 I | op-osd: OSD will have its main bluestore block on "smithi104-lv-4-data-0nmqwk" 2022-04-02T03:16:16.727 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:16:16.726113 I | op-osd: skipping osd checks for ok-to-stop 2022-04-02T03:16:16.842 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:16:16.840711 I | op-osd: updating OSD 3 on PVC "smithi142-lv-2-data-0qdbgg" 2022-04-02T03:16:16.842 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:16:16.840749 I | op-osd: OSD will have its main bluestore block on "smithi142-lv-2-data-0qdbgg" 2022-04-02T03:16:17.527 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:16:17.526170 I | op-osd: skipping osd checks for ok-to-stop 2022-04-02T03:16:17.621 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:16:17.619981 I | op-osd: updating OSD 4 on PVC "smithi142-lv-3-data-05pxj5" 2022-04-02T03:16:17.621 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:16:17.620019 I | op-osd: OSD will have its main bluestore block on "smithi142-lv-3-data-05pxj5" 2022-04-02T03:16:18.326 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:16:18.324456 I | op-osd: skipping osd checks for ok-to-stop 2022-04-02T03:16:18.421 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:16:18.419696 I | op-osd: updating OSD 5 on PVC "smithi029-lv-1-data-0xzkjz" 2022-04-02T03:16:18.421 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:16:18.419728 I | op-osd: OSD will have its main bluestore block on "smithi029-lv-1-data-0xzkjz" 2022-04-02T03:16:19.127 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:16:19.126175 I | op-osd: skipping osd checks for ok-to-stop 2022-04-02T03:16:19.220 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:16:19.218564 I | op-osd: updating OSD 6 on PVC "smithi142-lv-1-data-08v2jt" 2022-04-02T03:16:19.221 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:16:19.218600 I | op-osd: OSD will have its main bluestore block on "smithi142-lv-1-data-08v2jt" 2022-04-02T03:16:19.930 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:16:19.928666 I | op-osd: skipping osd checks for ok-to-stop 2022-04-02T03:16:20.020 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:16:20.018111 I | op-osd: updating OSD 7 on PVC "smithi029-lv-2-data-084vgd" 2022-04-02T03:16:20.020 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:16:20.018147 I | op-osd: OSD will have its main bluestore block on "smithi029-lv-2-data-084vgd" 2022-04-02T03:16:20.728 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:16:20.726863 I | op-osd: skipping osd checks for ok-to-stop 2022-04-02T03:16:20.821 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:16:20.819957 I | op-osd: updating OSD 8 on PVC "smithi142-lv-4-data-0hs4jx" 2022-04-02T03:16:20.822 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:16:20.819995 I | op-osd: OSD will have its main bluestore block on "smithi142-lv-4-data-0hs4jx" 2022-04-02T03:16:21.528 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:16:21.525463 I | op-osd: skipping osd checks for ok-to-stop 2022-04-02T03:16:21.558 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph osd stat -f json 2022-04-02T03:16:21.633 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:16:21.631542 I | op-osd: updating OSD 9 on PVC "smithi029-lv-4-data-0sxfm8" 2022-04-02T03:16:21.633 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:16:21.631578 I | op-osd: OSD will have its main bluestore block on "smithi029-lv-4-data-0sxfm8" 2022-04-02T03:16:22.089 INFO:teuthology.orchestra.run.smithi029.stdout: 2022-04-02T03:16:22.110 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":33,"num_osds":12,"num_up_osds":11,"osd_up_since":1648869375,"num_in_osds":12,"osd_in_since":1648869312,"num_remapped_pgs":0} 2022-04-02T03:16:22.111 INFO:tasks.rook: have 11/12 OSDs 2022-04-02T03:16:23.019 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:16:23.017017 I | op-osd: OSD orchestration status for node smithi104-lv-3-data-0jlrxt is "orchestrating" 2022-04-02T03:16:23.069 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:16:23.066886 I | op-osd: OSD orchestration status for PVC smithi104-lv-3-data-0jlrxt is "orchestrating" 2022-04-02T03:16:24.120 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:16:24.117800 I | op-osd: OSD orchestration status for PVC smithi104-lv-3-data-0jlrxt is "completed" 2022-04-02T03:16:24.120 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:16:24.117842 I | op-osd: creating OSD 11 on PVC "smithi104-lv-3-data-0jlrxt" 2022-04-02T03:16:24.120 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:16:24.117861 I | op-osd: OSD will have its main bluestore block on "smithi104-lv-3-data-0jlrxt" 2022-04-02T03:16:24.832 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:16:24.830065 I | op-osd: finished running OSDs in namespace "rook-ceph" 2022-04-02T03:16:24.833 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:16:24.830196 I | ceph-cluster-controller: done reconciling ceph cluster in namespace "rook-ceph" 2022-04-02T03:16:32.112 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph osd stat -f json 2022-04-02T03:16:32.668 INFO:teuthology.orchestra.run.smithi029.stdout: 2022-04-02T03:16:32.691 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":33,"num_osds":12,"num_up_osds":11,"osd_up_since":1648869375,"num_in_osds":12,"osd_in_since":1648869312,"num_remapped_pgs":0} 2022-04-02T03:16:32.692 INFO:tasks.rook: have 11/12 OSDs 2022-04-02T03:16:42.694 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph osd stat -f json 2022-04-02T03:16:43.213 INFO:teuthology.orchestra.run.smithi029.stdout: 2022-04-02T03:16:43.232 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":34,"num_osds":12,"num_up_osds":11,"osd_up_since":1648869375,"num_in_osds":12,"osd_in_since":1648869312,"num_remapped_pgs":0} 2022-04-02T03:16:43.233 INFO:tasks.rook: have 11/12 OSDs 2022-04-02T03:16:53.234 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph osd stat -f json 2022-04-02T03:16:53.784 INFO:teuthology.orchestra.run.smithi029.stdout: 2022-04-02T03:16:53.803 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":37,"num_osds":12,"num_up_osds":12,"osd_up_since":1648869404,"num_in_osds":12,"osd_in_since":1648869312,"num_remapped_pgs":0} 2022-04-02T03:16:53.803 INFO:tasks.rook:Distributing ceph config and client.admin keyring 2022-04-02T03:16:53.804 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- cat /etc/ceph/ceph.conf 2022-04-02T03:16:53.993 INFO:teuthology.orchestra.run.smithi029.stdout:[global] 2022-04-02T03:16:53.993 INFO:teuthology.orchestra.run.smithi029.stdout:mon_host = 10.102.133.199:6789,10.107.147.59:6789,10.96.208.36:6789 2022-04-02T03:16:53.993 INFO:teuthology.orchestra.run.smithi029.stdout: 2022-04-02T03:16:53.994 INFO:teuthology.orchestra.run.smithi029.stdout:[client.admin] 2022-04-02T03:16:53.994 INFO:teuthology.orchestra.run.smithi029.stdout:keyring = /etc/ceph/keyring 2022-04-02T03:16:53.997 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- cat /etc/ceph/keyring 2022-04-02T03:16:54.168 INFO:teuthology.orchestra.run.smithi029.stdout:[client.admin] 2022-04-02T03:16:54.169 INFO:teuthology.orchestra.run.smithi029.stdout:key = AQB/vkdixEA/AhAAg1CCYb5IzIAVi2OhSxuTkA== 2022-04-02T03:16:54.173 DEBUG:teuthology.orchestra.run.smithi029:> sudo mkdir -p /etc/ceph 2022-04-02T03:16:54.193 DEBUG:teuthology.orchestra.run.smithi104:> sudo mkdir -p /etc/ceph 2022-04-02T03:16:54.212 DEBUG:teuthology.orchestra.run.smithi142:> sudo mkdir -p /etc/ceph 2022-04-02T03:16:54.228 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2022-04-02T03:16:54.229 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/etc/ceph/ceph.conf 2022-04-02T03:16:54.244 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2022-04-02T03:16:54.245 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/etc/ceph/keyring 2022-04-02T03:16:54.302 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2022-04-02T03:16:54.303 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/etc/ceph/ceph.conf 2022-04-02T03:16:54.320 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2022-04-02T03:16:54.320 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/etc/ceph/keyring 2022-04-02T03:16:54.334 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2022-04-02T03:16:54.335 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd of=/etc/ceph/ceph.conf 2022-04-02T03:16:54.352 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2022-04-02T03:16:54.352 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd of=/etc/ceph/keyring 2022-04-02T03:16:54.407 INFO:tasks.rook:Setting up client nodes... 2022-04-02T03:16:54.408 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph auth get-or-create client.a mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2022-04-02T03:16:55.193 INFO:teuthology.orchestra.run.smithi029.stdout:[client.a] 2022-04-02T03:16:55.193 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQAnwEdiWWTABBAAoHxlMdXmn5hlZW4kzSFZSg== 2022-04-02T03:16:55.217 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2022-04-02T03:16:55.217 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/etc/ceph/ceph.client.a.keyring 2022-04-02T03:16:55.218 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 0644 /etc/ceph/ceph.client.a.keyring 2022-04-02T03:16:55.244 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph auth get-or-create client.b mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2022-04-02T03:16:56.060 INFO:teuthology.orchestra.run.smithi029.stdout:[client.b] 2022-04-02T03:16:56.061 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQAnwEdi3FgtORAAiK5oItMDWtn3/ss7MuZ5rg== 2022-04-02T03:16:56.086 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2022-04-02T03:16:56.086 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/etc/ceph/ceph.client.b.keyring 2022-04-02T03:16:56.087 DEBUG:teuthology.orchestra.run.smithi104:> sudo chmod 0644 /etc/ceph/ceph.client.b.keyring 2022-04-02T03:16:56.113 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph auth get-or-create client.c mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2022-04-02T03:16:56.861 INFO:teuthology.orchestra.run.smithi029.stdout:[client.c] 2022-04-02T03:16:56.861 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQAowEdiPdi4KxAAo/feKNIRjmY5f0Zq+GMlAQ== 2022-04-02T03:16:56.882 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2022-04-02T03:16:56.883 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd of=/etc/ceph/ceph.client.c.keyring 2022-04-02T03:16:56.883 DEBUG:teuthology.orchestra.run.smithi142:> sudo chmod 0644 /etc/ceph/ceph.client.c.keyring 2022-04-02T03:16:56.905 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph osd dump --format=json 2022-04-02T03:16:57.440 INFO:teuthology.orchestra.run.smithi029.stdout: 2022-04-02T03:16:57.440 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":37,"fsid":"17047c32-2dfc-4a19-8df4-d58d43ddcdff","created":"2022-04-02T03:11:28.410533+0000","modified":"2022-04-02T03:16:46.222621+0000","last_up_change":"2022-04-02T03:16:44.779668+0000","last_in_change":"2022-04-02T03:15:12.820882+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":15,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-04-02T03:15:46.757142+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"on","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"9710c818-b041-451e-b20a-d8686cdb08cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.226.14:6800","nonce":3407316437},{"type":"v1","addr":"10.248.226.14:6801","nonce":3407316437}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.226.14:6802","nonce":3407316437},{"type":"v1","addr":"10.248.226.14:6803","nonce":3407316437}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.226.14:6806","nonce":3407316437},{"type":"v1","addr":"10.248.226.14:6807","nonce":3407316437}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.226.14:6804","nonce":3407316437},{"type":"v1","addr":"10.248.226.14:6805","nonce":3407316437}]},"public_addr":"10.248.226.14:6801/3407316437","cluster_addr":"10.248.226.14:6803/3407316437","heartbeat_back_addr":"10.248.226.14:6807/3407316437","heartbeat_front_addr":"10.248.226.14:6805/3407316437","state":["exists","up"]},{"osd":1,"uuid":"c5a0c1b9-552f-4614-8341-19244a432aaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.226.15:6800","nonce":864073704},{"type":"v1","addr":"10.248.226.15:6801","nonce":864073704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.226.15:6802","nonce":864073704},{"type":"v1","addr":"10.248.226.15:6803","nonce":864073704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.226.15:6806","nonce":864073704},{"type":"v1","addr":"10.248.226.15:6807","nonce":864073704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.226.15:6804","nonce":864073704},{"type":"v1","addr":"10.248.226.15:6805","nonce":864073704}]},"public_addr":"10.248.226.15:6801/864073704","cluster_addr":"10.248.226.15:6803/864073704","heartbeat_back_addr":"10.248.226.15:6807/864073704","heartbeat_front_addr":"10.248.226.15:6805/864073704","state":["exists","up"]},{"osd":2,"uuid":"11a47675-c186-4ecf-8efd-5cdff9cf31c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.226.16:6800","nonce":386070420},{"type":"v1","addr":"10.248.226.16:6801","nonce":386070420}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.226.16:6802","nonce":386070420},{"type":"v1","addr":"10.248.226.16:6803","nonce":386070420}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.226.16:6806","nonce":386070420},{"type":"v1","addr":"10.248.226.16:6807","nonce":386070420}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.226.16:6804","nonce":386070420},{"type":"v1","addr":"10.248.226.16:6805","nonce":386070420}]},"public_addr":"10.248.226.16:6801/386070420","cluster_addr":"10.248.226.16:6803/386070420","heartbeat_back_addr":"10.248.226.16:6807/386070420","heartbeat_front_addr":"10.248.226.16:6805/386070420","state":["exists","up"]},{"osd":3,"uuid":"574ef572-a4c3-44e9-8d6b-0cf9313cd8d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.225.15:6800","nonce":2301310195},{"type":"v1","addr":"10.248.225.15:6801","nonce":2301310195}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.225.15:6802","nonce":2301310195},{"type":"v1","addr":"10.248.225.15:6803","nonce":2301310195}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.225.15:6806","nonce":2301310195},{"type":"v1","addr":"10.248.225.15:6807","nonce":2301310195}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.225.15:6804","nonce":2301310195},{"type":"v1","addr":"10.248.225.15:6805","nonce":2301310195}]},"public_addr":"10.248.225.15:6801/2301310195","cluster_addr":"10.248.225.15:6803/2301310195","heartbeat_back_addr":"10.248.225.15:6807/2301310195","heartbeat_front_addr":"10.248.225.15:6805/2301310195","state":["exists","up"]},{"osd":4,"uuid":"dc6c204a-d2e6-447e-9879-49ac9ff7afcd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.225.14:6800","nonce":3974197596},{"type":"v1","addr":"10.248.225.14:6801","nonce":3974197596}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.225.14:6802","nonce":3974197596},{"type":"v1","addr":"10.248.225.14:6803","nonce":3974197596}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.225.14:6806","nonce":3974197596},{"type":"v1","addr":"10.248.225.14:6807","nonce":3974197596}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.225.14:6804","nonce":3974197596},{"type":"v1","addr":"10.248.225.14:6805","nonce":3974197596}]},"public_addr":"10.248.225.14:6801/3974197596","cluster_addr":"10.248.225.14:6803/3974197596","heartbeat_back_addr":"10.248.225.14:6807/3974197596","heartbeat_front_addr":"10.248.225.14:6805/3974197596","state":["exists","up"]},{"osd":5,"uuid":"fee790e8-da28-4f8e-92a4-c8098642390e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":28,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.224.12:6800","nonce":1173881182},{"type":"v1","addr":"10.248.224.12:6801","nonce":1173881182}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.224.12:6802","nonce":1173881182},{"type":"v1","addr":"10.248.224.12:6803","nonce":1173881182}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.224.12:6806","nonce":1173881182},{"type":"v1","addr":"10.248.224.12:6807","nonce":1173881182}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.224.12:6804","nonce":1173881182},{"type":"v1","addr":"10.248.224.12:6805","nonce":1173881182}]},"public_addr":"10.248.224.12:6801/1173881182","cluster_addr":"10.248.224.12:6803/1173881182","heartbeat_back_addr":"10.248.224.12:6807/1173881182","heartbeat_front_addr":"10.248.224.12:6805/1173881182","state":["exists","up"]},{"osd":6,"uuid":"893ef865-bb3a-4746-b635-757f056c3014","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.225.16:6800","nonce":1100272466},{"type":"v1","addr":"10.248.225.16:6801","nonce":1100272466}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.225.16:6802","nonce":1100272466},{"type":"v1","addr":"10.248.225.16:6803","nonce":1100272466}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.225.16:6806","nonce":1100272466},{"type":"v1","addr":"10.248.225.16:6807","nonce":1100272466}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.225.16:6804","nonce":1100272466},{"type":"v1","addr":"10.248.225.16:6805","nonce":1100272466}]},"public_addr":"10.248.225.16:6801/1100272466","cluster_addr":"10.248.225.16:6803/1100272466","heartbeat_back_addr":"10.248.225.16:6807/1100272466","heartbeat_front_addr":"10.248.225.16:6805/1100272466","state":["exists","up"]},{"osd":7,"uuid":"e5cd3c7c-7486-4cfe-993b-566a3d04b01c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.224.14:6800","nonce":1357721307},{"type":"v1","addr":"10.248.224.14:6801","nonce":1357721307}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.224.14:6802","nonce":1357721307},{"type":"v1","addr":"10.248.224.14:6803","nonce":1357721307}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.224.14:6806","nonce":1357721307},{"type":"v1","addr":"10.248.224.14:6807","nonce":1357721307}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.224.14:6804","nonce":1357721307},{"type":"v1","addr":"10.248.224.14:6805","nonce":1357721307}]},"public_addr":"10.248.224.14:6801/1357721307","cluster_addr":"10.248.224.14:6803/1357721307","heartbeat_back_addr":"10.248.224.14:6807/1357721307","heartbeat_front_addr":"10.248.224.14:6805/1357721307","state":["exists","up"]},{"osd":8,"uuid":"85f16acc-1708-45b4-9b8b-790861c7cc58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.225.17:6800","nonce":615102439},{"type":"v1","addr":"10.248.225.17:6801","nonce":615102439}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.225.17:6802","nonce":615102439},{"type":"v1","addr":"10.248.225.17:6803","nonce":615102439}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.225.17:6806","nonce":615102439},{"type":"v1","addr":"10.248.225.17:6807","nonce":615102439}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.225.17:6804","nonce":615102439},{"type":"v1","addr":"10.248.225.17:6805","nonce":615102439}]},"public_addr":"10.248.225.17:6801/615102439","cluster_addr":"10.248.225.17:6803/615102439","heartbeat_back_addr":"10.248.225.17:6807/615102439","heartbeat_front_addr":"10.248.225.17:6805/615102439","state":["exists","up"]},{"osd":9,"uuid":"f97568fb-c60c-46e0-b6cf-841afb5bb79f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.224.15:6800","nonce":611587511},{"type":"v1","addr":"10.248.224.15:6801","nonce":611587511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.224.15:6802","nonce":611587511},{"type":"v1","addr":"10.248.224.15:6803","nonce":611587511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.224.15:6806","nonce":611587511},{"type":"v1","addr":"10.248.224.15:6807","nonce":611587511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.224.15:6804","nonce":611587511},{"type":"v1","addr":"10.248.224.15:6805","nonce":611587511}]},"public_addr":"10.248.224.15:6801/611587511","cluster_addr":"10.248.224.15:6803/611587511","heartbeat_back_addr":"10.248.224.15:6807/611587511","heartbeat_front_addr":"10.248.224.15:6805/611587511","state":["exists","up"]},{"osd":10,"uuid":"927e067b-a075-416e-9cfc-f54167ad1fbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.224.16:6800","nonce":2762218126},{"type":"v1","addr":"10.248.224.16:6801","nonce":2762218126}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.224.16:6802","nonce":2762218126},{"type":"v1","addr":"10.248.224.16:6803","nonce":2762218126}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.224.16:6806","nonce":2762218126},{"type":"v1","addr":"10.248.224.16:6807","nonce":2762218126}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.224.16:6804","nonce":2762218126},{"type":"v1","addr":"10.248.224.16:6805","nonce":2762218126}]},"public_addr":"10.248.224.16:6801/2762218126","cluster_addr":"10.248.224.16:6803/2762218126","heartbeat_back_addr":"10.248.224.16:6807/2762218126","heartbeat_front_addr":"10.248.224.16:6805/2762218126","state":["exists","up"]},{"osd":11,"uuid":"290b1ffa-2b9a-48cb-bd52-c0b18052f06c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.226.18:6800","nonce":1227626749},{"type":"v1","addr":"10.248.226.18:6801","nonce":1227626749}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.226.18:6802","nonce":1227626749},{"type":"v1","addr":"10.248.226.18:6803","nonce":1227626749}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.226.18:6806","nonce":1227626749},{"type":"v1","addr":"10.248.226.18:6807","nonce":1227626749}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.226.18:6804","nonce":1227626749},{"type":"v1","addr":"10.248.226.18:6805","nonce":1227626749}]},"public_addr":"10.248.226.18:6801/1227626749","cluster_addr":"10.248.226.18:6803/1227626749","heartbeat_back_addr":"10.248.226.18:6807/1227626749","heartbeat_front_addr":"10.248.226.18:6805/1227626749","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-02T03:15:33.857398+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-02T03:15:37.545150+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-02T03:15:37.465996+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-02T03:15:39.084681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-02T03:15:37.557451+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-02T03:16:07.932933+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-02T03:15:44.332659+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-02T03:16:07.614231+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-02T03:15:43.520036+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-02T03:16:12.931996+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-02T03:16:13.099469+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-02T03:16:42.635049+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.248.224.4:6800/4020328410":"2022-04-03T03:13:50.530591+0000","10.248.224.4:6801/4020328410":"2022-04-03T03:13:50.530591+0000","10.248.224.4:0/1305699235":"2022-04-03T03:13:50.530591+0000","10.248.224.4:0/1916824385":"2022-04-03T03:13:50.530591+0000","10.248.224.4:0/2181722632":"2022-04-03T03:13:50.530591+0000"},"erasure_code_profiles":{"default":{"k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-04-02T03:16:57.460 INFO:tasks.rook.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2022-04-02T03:15:46.757142+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'on', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2022-04-02T03:16:57.461 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph osd pool get .mgr pg_num 2022-04-02T03:16:57.970 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2022-04-02T03:16:57.993 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2022-04-02T03:16:57.995 INFO:tasks.rook.ceph_manager.ceph:waiting for mgr available 2022-04-02T03:16:57.995 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph mgr dump --format=json 2022-04-02T03:16:58.570 INFO:teuthology.orchestra.run.smithi029.stdout: 2022-04-02T03:16:58.591 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":14,"active_gid":4335,"active_name":"a","active_addrs":{"addrvec":[{"type":"v2","addr":"10.248.224.4:6800","nonce":3959457987},{"type":"v1","addr":"10.248.224.4:6801","nonce":3959457987}]},"active_addr":"10.248.224.4:6801/3959457987","active_change":"2022-04-02T03:13:50.559403+0000","active_mgr_features":4540138303579357183,"available":true,"standbys":[],"modules":["iostat","nfs","prometheus","restful","rook"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:6.7.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"0.0.0.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"0.0.0.0","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"prometheus":"http://10.248.224.4:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":2,"active_clients":[{"addrvec":[{"type":"v2","addr":"10.248.224.4:0","nonce":53764737}]},{"addrvec":[{"type":"v2","addr":"10.248.224.4:0","nonce":2470427612}]},{"addrvec":[{"type":"v2","addr":"10.248.224.4:0","nonce":3994214949}]},{"addrvec":[{"type":"v2","addr":"10.248.224.4:0","nonce":2751186980}]}]}} 2022-04-02T03:16:58.594 INFO:tasks.rook.ceph_manager.ceph:mgr available! 2022-04-02T03:16:58.594 INFO:tasks.rook.ceph_manager.ceph:waiting for all up 2022-04-02T03:16:58.595 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph osd dump --format=json 2022-04-02T03:16:59.101 INFO:teuthology.orchestra.run.smithi029.stdout: 2022-04-02T03:16:59.101 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":37,"fsid":"17047c32-2dfc-4a19-8df4-d58d43ddcdff","created":"2022-04-02T03:11:28.410533+0000","modified":"2022-04-02T03:16:46.222621+0000","last_up_change":"2022-04-02T03:16:44.779668+0000","last_in_change":"2022-04-02T03:15:12.820882+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":15,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-04-02T03:15:46.757142+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"on","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"9710c818-b041-451e-b20a-d8686cdb08cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.226.14:6800","nonce":3407316437},{"type":"v1","addr":"10.248.226.14:6801","nonce":3407316437}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.226.14:6802","nonce":3407316437},{"type":"v1","addr":"10.248.226.14:6803","nonce":3407316437}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.226.14:6806","nonce":3407316437},{"type":"v1","addr":"10.248.226.14:6807","nonce":3407316437}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.226.14:6804","nonce":3407316437},{"type":"v1","addr":"10.248.226.14:6805","nonce":3407316437}]},"public_addr":"10.248.226.14:6801/3407316437","cluster_addr":"10.248.226.14:6803/3407316437","heartbeat_back_addr":"10.248.226.14:6807/3407316437","heartbeat_front_addr":"10.248.226.14:6805/3407316437","state":["exists","up"]},{"osd":1,"uuid":"c5a0c1b9-552f-4614-8341-19244a432aaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.226.15:6800","nonce":864073704},{"type":"v1","addr":"10.248.226.15:6801","nonce":864073704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.226.15:6802","nonce":864073704},{"type":"v1","addr":"10.248.226.15:6803","nonce":864073704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.226.15:6806","nonce":864073704},{"type":"v1","addr":"10.248.226.15:6807","nonce":864073704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.226.15:6804","nonce":864073704},{"type":"v1","addr":"10.248.226.15:6805","nonce":864073704}]},"public_addr":"10.248.226.15:6801/864073704","cluster_addr":"10.248.226.15:6803/864073704","heartbeat_back_addr":"10.248.226.15:6807/864073704","heartbeat_front_addr":"10.248.226.15:6805/864073704","state":["exists","up"]},{"osd":2,"uuid":"11a47675-c186-4ecf-8efd-5cdff9cf31c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.226.16:6800","nonce":386070420},{"type":"v1","addr":"10.248.226.16:6801","nonce":386070420}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.226.16:6802","nonce":386070420},{"type":"v1","addr":"10.248.226.16:6803","nonce":386070420}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.226.16:6806","nonce":386070420},{"type":"v1","addr":"10.248.226.16:6807","nonce":386070420}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.226.16:6804","nonce":386070420},{"type":"v1","addr":"10.248.226.16:6805","nonce":386070420}]},"public_addr":"10.248.226.16:6801/386070420","cluster_addr":"10.248.226.16:6803/386070420","heartbeat_back_addr":"10.248.226.16:6807/386070420","heartbeat_front_addr":"10.248.226.16:6805/386070420","state":["exists","up"]},{"osd":3,"uuid":"574ef572-a4c3-44e9-8d6b-0cf9313cd8d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.225.15:6800","nonce":2301310195},{"type":"v1","addr":"10.248.225.15:6801","nonce":2301310195}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.225.15:6802","nonce":2301310195},{"type":"v1","addr":"10.248.225.15:6803","nonce":2301310195}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.225.15:6806","nonce":2301310195},{"type":"v1","addr":"10.248.225.15:6807","nonce":2301310195}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.225.15:6804","nonce":2301310195},{"type":"v1","addr":"10.248.225.15:6805","nonce":2301310195}]},"public_addr":"10.248.225.15:6801/2301310195","cluster_addr":"10.248.225.15:6803/2301310195","heartbeat_back_addr":"10.248.225.15:6807/2301310195","heartbeat_front_addr":"10.248.225.15:6805/2301310195","state":["exists","up"]},{"osd":4,"uuid":"dc6c204a-d2e6-447e-9879-49ac9ff7afcd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.225.14:6800","nonce":3974197596},{"type":"v1","addr":"10.248.225.14:6801","nonce":3974197596}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.225.14:6802","nonce":3974197596},{"type":"v1","addr":"10.248.225.14:6803","nonce":3974197596}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.225.14:6806","nonce":3974197596},{"type":"v1","addr":"10.248.225.14:6807","nonce":3974197596}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.225.14:6804","nonce":3974197596},{"type":"v1","addr":"10.248.225.14:6805","nonce":3974197596}]},"public_addr":"10.248.225.14:6801/3974197596","cluster_addr":"10.248.225.14:6803/3974197596","heartbeat_back_addr":"10.248.225.14:6807/3974197596","heartbeat_front_addr":"10.248.225.14:6805/3974197596","state":["exists","up"]},{"osd":5,"uuid":"fee790e8-da28-4f8e-92a4-c8098642390e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":28,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.224.12:6800","nonce":1173881182},{"type":"v1","addr":"10.248.224.12:6801","nonce":1173881182}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.224.12:6802","nonce":1173881182},{"type":"v1","addr":"10.248.224.12:6803","nonce":1173881182}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.224.12:6806","nonce":1173881182},{"type":"v1","addr":"10.248.224.12:6807","nonce":1173881182}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.224.12:6804","nonce":1173881182},{"type":"v1","addr":"10.248.224.12:6805","nonce":1173881182}]},"public_addr":"10.248.224.12:6801/1173881182","cluster_addr":"10.248.224.12:6803/1173881182","heartbeat_back_addr":"10.248.224.12:6807/1173881182","heartbeat_front_addr":"10.248.224.12:6805/1173881182","state":["exists","up"]},{"osd":6,"uuid":"893ef865-bb3a-4746-b635-757f056c3014","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.225.16:6800","nonce":1100272466},{"type":"v1","addr":"10.248.225.16:6801","nonce":1100272466}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.225.16:6802","nonce":1100272466},{"type":"v1","addr":"10.248.225.16:6803","nonce":1100272466}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.225.16:6806","nonce":1100272466},{"type":"v1","addr":"10.248.225.16:6807","nonce":1100272466}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.225.16:6804","nonce":1100272466},{"type":"v1","addr":"10.248.225.16:6805","nonce":1100272466}]},"public_addr":"10.248.225.16:6801/1100272466","cluster_addr":"10.248.225.16:6803/1100272466","heartbeat_back_addr":"10.248.225.16:6807/1100272466","heartbeat_front_addr":"10.248.225.16:6805/1100272466","state":["exists","up"]},{"osd":7,"uuid":"e5cd3c7c-7486-4cfe-993b-566a3d04b01c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.224.14:6800","nonce":1357721307},{"type":"v1","addr":"10.248.224.14:6801","nonce":1357721307}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.224.14:6802","nonce":1357721307},{"type":"v1","addr":"10.248.224.14:6803","nonce":1357721307}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.224.14:6806","nonce":1357721307},{"type":"v1","addr":"10.248.224.14:6807","nonce":1357721307}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.224.14:6804","nonce":1357721307},{"type":"v1","addr":"10.248.224.14:6805","nonce":1357721307}]},"public_addr":"10.248.224.14:6801/1357721307","cluster_addr":"10.248.224.14:6803/1357721307","heartbeat_back_addr":"10.248.224.14:6807/1357721307","heartbeat_front_addr":"10.248.224.14:6805/1357721307","state":["exists","up"]},{"osd":8,"uuid":"85f16acc-1708-45b4-9b8b-790861c7cc58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.225.17:6800","nonce":615102439},{"type":"v1","addr":"10.248.225.17:6801","nonce":615102439}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.225.17:6802","nonce":615102439},{"type":"v1","addr":"10.248.225.17:6803","nonce":615102439}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.225.17:6806","nonce":615102439},{"type":"v1","addr":"10.248.225.17:6807","nonce":615102439}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.225.17:6804","nonce":615102439},{"type":"v1","addr":"10.248.225.17:6805","nonce":615102439}]},"public_addr":"10.248.225.17:6801/615102439","cluster_addr":"10.248.225.17:6803/615102439","heartbeat_back_addr":"10.248.225.17:6807/615102439","heartbeat_front_addr":"10.248.225.17:6805/615102439","state":["exists","up"]},{"osd":9,"uuid":"f97568fb-c60c-46e0-b6cf-841afb5bb79f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.224.15:6800","nonce":611587511},{"type":"v1","addr":"10.248.224.15:6801","nonce":611587511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.224.15:6802","nonce":611587511},{"type":"v1","addr":"10.248.224.15:6803","nonce":611587511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.224.15:6806","nonce":611587511},{"type":"v1","addr":"10.248.224.15:6807","nonce":611587511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.224.15:6804","nonce":611587511},{"type":"v1","addr":"10.248.224.15:6805","nonce":611587511}]},"public_addr":"10.248.224.15:6801/611587511","cluster_addr":"10.248.224.15:6803/611587511","heartbeat_back_addr":"10.248.224.15:6807/611587511","heartbeat_front_addr":"10.248.224.15:6805/611587511","state":["exists","up"]},{"osd":10,"uuid":"927e067b-a075-416e-9cfc-f54167ad1fbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.224.16:6800","nonce":2762218126},{"type":"v1","addr":"10.248.224.16:6801","nonce":2762218126}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.224.16:6802","nonce":2762218126},{"type":"v1","addr":"10.248.224.16:6803","nonce":2762218126}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.224.16:6806","nonce":2762218126},{"type":"v1","addr":"10.248.224.16:6807","nonce":2762218126}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.224.16:6804","nonce":2762218126},{"type":"v1","addr":"10.248.224.16:6805","nonce":2762218126}]},"public_addr":"10.248.224.16:6801/2762218126","cluster_addr":"10.248.224.16:6803/2762218126","heartbeat_back_addr":"10.248.224.16:6807/2762218126","heartbeat_front_addr":"10.248.224.16:6805/2762218126","state":["exists","up"]},{"osd":11,"uuid":"290b1ffa-2b9a-48cb-bd52-c0b18052f06c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.226.18:6800","nonce":1227626749},{"type":"v1","addr":"10.248.226.18:6801","nonce":1227626749}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.226.18:6802","nonce":1227626749},{"type":"v1","addr":"10.248.226.18:6803","nonce":1227626749}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.226.18:6806","nonce":1227626749},{"type":"v1","addr":"10.248.226.18:6807","nonce":1227626749}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.226.18:6804","nonce":1227626749},{"type":"v1","addr":"10.248.226.18:6805","nonce":1227626749}]},"public_addr":"10.248.226.18:6801/1227626749","cluster_addr":"10.248.226.18:6803/1227626749","heartbeat_back_addr":"10.248.226.18:6807/1227626749","heartbeat_front_addr":"10.248.226.18:6805/1227626749","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-02T03:15:33.857398+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-02T03:15:37.545150+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-02T03:15:37.465996+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-02T03:15:39.084681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-02T03:15:37.557451+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-02T03:16:07.932933+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-02T03:15:44.332659+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-02T03:16:07.614231+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-02T03:15:43.520036+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-02T03:16:12.931996+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-02T03:16:13.099469+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-02T03:16:42.635049+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.248.224.4:6800/4020328410":"2022-04-03T03:13:50.530591+0000","10.248.224.4:6801/4020328410":"2022-04-03T03:13:50.530591+0000","10.248.224.4:0/1305699235":"2022-04-03T03:13:50.530591+0000","10.248.224.4:0/1916824385":"2022-04-03T03:13:50.530591+0000","10.248.224.4:0/2181722632":"2022-04-03T03:13:50.530591+0000"},"erasure_code_profiles":{"default":{"k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-04-02T03:16:59.119 INFO:tasks.rook.ceph_manager.ceph:all up! 2022-04-02T03:16:59.119 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph osd dump --format=json 2022-04-02T03:16:59.618 INFO:teuthology.orchestra.run.smithi029.stdout: 2022-04-02T03:16:59.618 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":37,"fsid":"17047c32-2dfc-4a19-8df4-d58d43ddcdff","created":"2022-04-02T03:11:28.410533+0000","modified":"2022-04-02T03:16:46.222621+0000","last_up_change":"2022-04-02T03:16:44.779668+0000","last_in_change":"2022-04-02T03:15:12.820882+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":15,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-04-02T03:15:46.757142+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"on","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"9710c818-b041-451e-b20a-d8686cdb08cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.226.14:6800","nonce":3407316437},{"type":"v1","addr":"10.248.226.14:6801","nonce":3407316437}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.226.14:6802","nonce":3407316437},{"type":"v1","addr":"10.248.226.14:6803","nonce":3407316437}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.226.14:6806","nonce":3407316437},{"type":"v1","addr":"10.248.226.14:6807","nonce":3407316437}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.226.14:6804","nonce":3407316437},{"type":"v1","addr":"10.248.226.14:6805","nonce":3407316437}]},"public_addr":"10.248.226.14:6801/3407316437","cluster_addr":"10.248.226.14:6803/3407316437","heartbeat_back_addr":"10.248.226.14:6807/3407316437","heartbeat_front_addr":"10.248.226.14:6805/3407316437","state":["exists","up"]},{"osd":1,"uuid":"c5a0c1b9-552f-4614-8341-19244a432aaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.226.15:6800","nonce":864073704},{"type":"v1","addr":"10.248.226.15:6801","nonce":864073704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.226.15:6802","nonce":864073704},{"type":"v1","addr":"10.248.226.15:6803","nonce":864073704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.226.15:6806","nonce":864073704},{"type":"v1","addr":"10.248.226.15:6807","nonce":864073704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.226.15:6804","nonce":864073704},{"type":"v1","addr":"10.248.226.15:6805","nonce":864073704}]},"public_addr":"10.248.226.15:6801/864073704","cluster_addr":"10.248.226.15:6803/864073704","heartbeat_back_addr":"10.248.226.15:6807/864073704","heartbeat_front_addr":"10.248.226.15:6805/864073704","state":["exists","up"]},{"osd":2,"uuid":"11a47675-c186-4ecf-8efd-5cdff9cf31c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.226.16:6800","nonce":386070420},{"type":"v1","addr":"10.248.226.16:6801","nonce":386070420}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.226.16:6802","nonce":386070420},{"type":"v1","addr":"10.248.226.16:6803","nonce":386070420}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.226.16:6806","nonce":386070420},{"type":"v1","addr":"10.248.226.16:6807","nonce":386070420}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.226.16:6804","nonce":386070420},{"type":"v1","addr":"10.248.226.16:6805","nonce":386070420}]},"public_addr":"10.248.226.16:6801/386070420","cluster_addr":"10.248.226.16:6803/386070420","heartbeat_back_addr":"10.248.226.16:6807/386070420","heartbeat_front_addr":"10.248.226.16:6805/386070420","state":["exists","up"]},{"osd":3,"uuid":"574ef572-a4c3-44e9-8d6b-0cf9313cd8d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.225.15:6800","nonce":2301310195},{"type":"v1","addr":"10.248.225.15:6801","nonce":2301310195}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.225.15:6802","nonce":2301310195},{"type":"v1","addr":"10.248.225.15:6803","nonce":2301310195}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.225.15:6806","nonce":2301310195},{"type":"v1","addr":"10.248.225.15:6807","nonce":2301310195}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.225.15:6804","nonce":2301310195},{"type":"v1","addr":"10.248.225.15:6805","nonce":2301310195}]},"public_addr":"10.248.225.15:6801/2301310195","cluster_addr":"10.248.225.15:6803/2301310195","heartbeat_back_addr":"10.248.225.15:6807/2301310195","heartbeat_front_addr":"10.248.225.15:6805/2301310195","state":["exists","up"]},{"osd":4,"uuid":"dc6c204a-d2e6-447e-9879-49ac9ff7afcd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.225.14:6800","nonce":3974197596},{"type":"v1","addr":"10.248.225.14:6801","nonce":3974197596}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.225.14:6802","nonce":3974197596},{"type":"v1","addr":"10.248.225.14:6803","nonce":3974197596}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.225.14:6806","nonce":3974197596},{"type":"v1","addr":"10.248.225.14:6807","nonce":3974197596}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.225.14:6804","nonce":3974197596},{"type":"v1","addr":"10.248.225.14:6805","nonce":3974197596}]},"public_addr":"10.248.225.14:6801/3974197596","cluster_addr":"10.248.225.14:6803/3974197596","heartbeat_back_addr":"10.248.225.14:6807/3974197596","heartbeat_front_addr":"10.248.225.14:6805/3974197596","state":["exists","up"]},{"osd":5,"uuid":"fee790e8-da28-4f8e-92a4-c8098642390e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":28,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.224.12:6800","nonce":1173881182},{"type":"v1","addr":"10.248.224.12:6801","nonce":1173881182}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.224.12:6802","nonce":1173881182},{"type":"v1","addr":"10.248.224.12:6803","nonce":1173881182}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.224.12:6806","nonce":1173881182},{"type":"v1","addr":"10.248.224.12:6807","nonce":1173881182}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.224.12:6804","nonce":1173881182},{"type":"v1","addr":"10.248.224.12:6805","nonce":1173881182}]},"public_addr":"10.248.224.12:6801/1173881182","cluster_addr":"10.248.224.12:6803/1173881182","heartbeat_back_addr":"10.248.224.12:6807/1173881182","heartbeat_front_addr":"10.248.224.12:6805/1173881182","state":["exists","up"]},{"osd":6,"uuid":"893ef865-bb3a-4746-b635-757f056c3014","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.225.16:6800","nonce":1100272466},{"type":"v1","addr":"10.248.225.16:6801","nonce":1100272466}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.225.16:6802","nonce":1100272466},{"type":"v1","addr":"10.248.225.16:6803","nonce":1100272466}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.225.16:6806","nonce":1100272466},{"type":"v1","addr":"10.248.225.16:6807","nonce":1100272466}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.225.16:6804","nonce":1100272466},{"type":"v1","addr":"10.248.225.16:6805","nonce":1100272466}]},"public_addr":"10.248.225.16:6801/1100272466","cluster_addr":"10.248.225.16:6803/1100272466","heartbeat_back_addr":"10.248.225.16:6807/1100272466","heartbeat_front_addr":"10.248.225.16:6805/1100272466","state":["exists","up"]},{"osd":7,"uuid":"e5cd3c7c-7486-4cfe-993b-566a3d04b01c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.224.14:6800","nonce":1357721307},{"type":"v1","addr":"10.248.224.14:6801","nonce":1357721307}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.224.14:6802","nonce":1357721307},{"type":"v1","addr":"10.248.224.14:6803","nonce":1357721307}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.224.14:6806","nonce":1357721307},{"type":"v1","addr":"10.248.224.14:6807","nonce":1357721307}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.224.14:6804","nonce":1357721307},{"type":"v1","addr":"10.248.224.14:6805","nonce":1357721307}]},"public_addr":"10.248.224.14:6801/1357721307","cluster_addr":"10.248.224.14:6803/1357721307","heartbeat_back_addr":"10.248.224.14:6807/1357721307","heartbeat_front_addr":"10.248.224.14:6805/1357721307","state":["exists","up"]},{"osd":8,"uuid":"85f16acc-1708-45b4-9b8b-790861c7cc58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.225.17:6800","nonce":615102439},{"type":"v1","addr":"10.248.225.17:6801","nonce":615102439}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.225.17:6802","nonce":615102439},{"type":"v1","addr":"10.248.225.17:6803","nonce":615102439}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.225.17:6806","nonce":615102439},{"type":"v1","addr":"10.248.225.17:6807","nonce":615102439}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.225.17:6804","nonce":615102439},{"type":"v1","addr":"10.248.225.17:6805","nonce":615102439}]},"public_addr":"10.248.225.17:6801/615102439","cluster_addr":"10.248.225.17:6803/615102439","heartbeat_back_addr":"10.248.225.17:6807/615102439","heartbeat_front_addr":"10.248.225.17:6805/615102439","state":["exists","up"]},{"osd":9,"uuid":"f97568fb-c60c-46e0-b6cf-841afb5bb79f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.224.15:6800","nonce":611587511},{"type":"v1","addr":"10.248.224.15:6801","nonce":611587511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.224.15:6802","nonce":611587511},{"type":"v1","addr":"10.248.224.15:6803","nonce":611587511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.224.15:6806","nonce":611587511},{"type":"v1","addr":"10.248.224.15:6807","nonce":611587511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.224.15:6804","nonce":611587511},{"type":"v1","addr":"10.248.224.15:6805","nonce":611587511}]},"public_addr":"10.248.224.15:6801/611587511","cluster_addr":"10.248.224.15:6803/611587511","heartbeat_back_addr":"10.248.224.15:6807/611587511","heartbeat_front_addr":"10.248.224.15:6805/611587511","state":["exists","up"]},{"osd":10,"uuid":"927e067b-a075-416e-9cfc-f54167ad1fbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.224.16:6800","nonce":2762218126},{"type":"v1","addr":"10.248.224.16:6801","nonce":2762218126}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.224.16:6802","nonce":2762218126},{"type":"v1","addr":"10.248.224.16:6803","nonce":2762218126}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.224.16:6806","nonce":2762218126},{"type":"v1","addr":"10.248.224.16:6807","nonce":2762218126}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.224.16:6804","nonce":2762218126},{"type":"v1","addr":"10.248.224.16:6805","nonce":2762218126}]},"public_addr":"10.248.224.16:6801/2762218126","cluster_addr":"10.248.224.16:6803/2762218126","heartbeat_back_addr":"10.248.224.16:6807/2762218126","heartbeat_front_addr":"10.248.224.16:6805/2762218126","state":["exists","up"]},{"osd":11,"uuid":"290b1ffa-2b9a-48cb-bd52-c0b18052f06c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.226.18:6800","nonce":1227626749},{"type":"v1","addr":"10.248.226.18:6801","nonce":1227626749}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.226.18:6802","nonce":1227626749},{"type":"v1","addr":"10.248.226.18:6803","nonce":1227626749}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.226.18:6806","nonce":1227626749},{"type":"v1","addr":"10.248.226.18:6807","nonce":1227626749}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.226.18:6804","nonce":1227626749},{"type":"v1","addr":"10.248.226.18:6805","nonce":1227626749}]},"public_addr":"10.248.226.18:6801/1227626749","cluster_addr":"10.248.226.18:6803/1227626749","heartbeat_back_addr":"10.248.226.18:6807/1227626749","heartbeat_front_addr":"10.248.226.18:6805/1227626749","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-02T03:15:33.857398+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-02T03:15:37.545150+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-02T03:15:37.465996+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-02T03:15:39.084681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-02T03:15:37.557451+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-02T03:16:07.932933+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-02T03:15:44.332659+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-02T03:16:07.614231+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-02T03:15:43.520036+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-02T03:16:12.931996+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-02T03:16:13.099469+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-02T03:16:42.635049+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.248.224.4:6800/4020328410":"2022-04-03T03:13:50.530591+0000","10.248.224.4:6801/4020328410":"2022-04-03T03:13:50.530591+0000","10.248.224.4:0/1305699235":"2022-04-03T03:13:50.530591+0000","10.248.224.4:0/1916824385":"2022-04-03T03:13:50.530591+0000","10.248.224.4:0/2181722632":"2022-04-03T03:13:50.530591+0000"},"erasure_code_profiles":{"default":{"k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-04-02T03:16:59.639 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph tell osd.0 flush_pg_stats 2022-04-02T03:16:59.640 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph tell osd.1 flush_pg_stats 2022-04-02T03:16:59.640 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph tell osd.2 flush_pg_stats 2022-04-02T03:16:59.641 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph tell osd.3 flush_pg_stats 2022-04-02T03:16:59.642 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph tell osd.4 flush_pg_stats 2022-04-02T03:16:59.643 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph tell osd.5 flush_pg_stats 2022-04-02T03:16:59.645 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph tell osd.6 flush_pg_stats 2022-04-02T03:16:59.646 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph tell osd.7 flush_pg_stats 2022-04-02T03:16:59.646 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph tell osd.8 flush_pg_stats 2022-04-02T03:16:59.647 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph tell osd.9 flush_pg_stats 2022-04-02T03:16:59.648 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph tell osd.10 flush_pg_stats 2022-04-02T03:16:59.648 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph tell osd.11 flush_pg_stats 2022-04-02T03:17:00.178 INFO:teuthology.orchestra.run.smithi029.stdout:120259084300 2022-04-02T03:17:00.179 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph osd last-stat-seq osd.5 2022-04-02T03:17:00.496 INFO:teuthology.orchestra.run.smithi029.stdout:81604378643 2022-04-02T03:17:00.497 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph osd last-stat-seq osd.1 2022-04-02T03:17:00.499 INFO:teuthology.orchestra.run.smithi029.stdout:137438953484 2022-04-02T03:17:00.500 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph osd last-stat-seq osd.9 2022-04-02T03:17:00.502 INFO:teuthology.orchestra.run.smithi029.stdout:94489280529 2022-04-02T03:17:00.503 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph osd last-stat-seq osd.8 2022-04-02T03:17:00.504 INFO:teuthology.orchestra.run.smithi029.stdout:73014444052 2022-04-02T03:17:00.505 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph osd last-stat-seq osd.0 2022-04-02T03:17:00.509 INFO:teuthology.orchestra.run.smithi029.stdout:81604378643 2022-04-02T03:17:00.510 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph osd last-stat-seq osd.2 2022-04-02T03:17:00.524 INFO:teuthology.orchestra.run.smithi029.stdout:154618822662 2022-04-02T03:17:00.524 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph osd last-stat-seq osd.11 2022-04-02T03:17:00.529 INFO:teuthology.orchestra.run.smithi029.stdout:94489280530 2022-04-02T03:17:00.530 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph osd last-stat-seq osd.3 2022-04-02T03:17:00.546 INFO:teuthology.orchestra.run.smithi029.stdout:85899345941 2022-04-02T03:17:00.547 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph osd last-stat-seq osd.4 2022-04-02T03:17:00.590 INFO:teuthology.orchestra.run.smithi029.stdout:98784247826 2022-04-02T03:17:00.591 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph osd last-stat-seq osd.6 2022-04-02T03:17:00.708 INFO:teuthology.orchestra.run.smithi029.stdout:137438953483 2022-04-02T03:17:00.708 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph osd last-stat-seq osd.10 2022-04-02T03:17:00.710 INFO:teuthology.orchestra.run.smithi029.stdout:115964117005 2022-04-02T03:17:00.711 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph osd last-stat-seq osd.7 2022-04-02T03:17:01.035 INFO:teuthology.orchestra.run.smithi029.stdout:120259084299 2022-04-02T03:17:01.124 INFO:tasks.rook.ceph_manager.ceph:need seq 120259084300 got 120259084299 for osd.5 2022-04-02T03:17:01.365 INFO:teuthology.orchestra.run.smithi029.stdout:81604378642 2022-04-02T03:17:01.381 INFO:teuthology.orchestra.run.smithi029.stdout:137438953483 2022-04-02T03:17:01.439 INFO:teuthology.orchestra.run.smithi029.stdout:73014444050 2022-04-02T03:17:01.528 INFO:tasks.rook.ceph_manager.ceph:need seq 81604378643 got 81604378642 for osd.2 2022-04-02T03:17:01.531 INFO:tasks.rook.ceph_manager.ceph:need seq 137438953484 got 137438953483 for osd.9 2022-04-02T03:17:01.536 INFO:tasks.rook.ceph_manager.ceph:need seq 73014444052 got 73014444050 for osd.0 2022-04-02T03:17:01.544 INFO:teuthology.orchestra.run.smithi029.stdout:81604378642 2022-04-02T03:17:01.556 INFO:teuthology.orchestra.run.smithi029.stdout:94489280528 2022-04-02T03:17:01.570 INFO:tasks.rook.ceph_manager.ceph:need seq 81604378643 got 81604378642 for osd.1 2022-04-02T03:17:01.598 INFO:tasks.rook.ceph_manager.ceph:need seq 94489280529 got 94489280528 for osd.8 2022-04-02T03:17:01.626 INFO:teuthology.orchestra.run.smithi029.stdout:98784247824 2022-04-02T03:17:01.645 INFO:tasks.rook.ceph_manager.ceph:need seq 98784247826 got 98784247824 for osd.6 2022-04-02T03:17:01.723 INFO:teuthology.orchestra.run.smithi029.stdout:94489280530 2022-04-02T03:17:01.747 INFO:tasks.rook.ceph_manager.ceph:need seq 94489280530 got 94489280530 for osd.3 2022-04-02T03:17:01.747 DEBUG:teuthology.parallel:result is None 2022-04-02T03:17:01.817 INFO:teuthology.orchestra.run.smithi029.stdout:154618822663 2022-04-02T03:17:01.820 INFO:teuthology.orchestra.run.smithi029.stdout:85899345941 2022-04-02T03:17:01.824 INFO:teuthology.orchestra.run.smithi029.stdout:137438953483 2022-04-02T03:17:01.844 INFO:tasks.rook.ceph_manager.ceph:need seq 154618822662 got 154618822663 for osd.11 2022-04-02T03:17:01.845 DEBUG:teuthology.parallel:result is None 2022-04-02T03:17:01.848 INFO:tasks.rook.ceph_manager.ceph:need seq 85899345941 got 85899345941 for osd.4 2022-04-02T03:17:01.848 DEBUG:teuthology.parallel:result is None 2022-04-02T03:17:01.850 INFO:tasks.rook.ceph_manager.ceph:need seq 137438953483 got 137438953483 for osd.10 2022-04-02T03:17:01.850 DEBUG:teuthology.parallel:result is None 2022-04-02T03:17:01.941 INFO:teuthology.orchestra.run.smithi029.stdout:115964117005 2022-04-02T03:17:01.960 INFO:tasks.rook.ceph_manager.ceph:need seq 115964117005 got 115964117005 for osd.7 2022-04-02T03:17:01.961 DEBUG:teuthology.parallel:result is None 2022-04-02T03:17:02.125 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph osd last-stat-seq osd.5 2022-04-02T03:17:02.529 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph osd last-stat-seq osd.2 2022-04-02T03:17:02.532 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph osd last-stat-seq osd.9 2022-04-02T03:17:02.537 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph osd last-stat-seq osd.0 2022-04-02T03:17:02.571 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph osd last-stat-seq osd.1 2022-04-02T03:17:02.600 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph osd last-stat-seq osd.8 2022-04-02T03:17:02.643 INFO:teuthology.orchestra.run.smithi029.stdout:120259084300 2022-04-02T03:17:02.647 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph osd last-stat-seq osd.6 2022-04-02T03:17:02.691 INFO:tasks.rook.ceph_manager.ceph:need seq 120259084300 got 120259084300 for osd.5 2022-04-02T03:17:02.691 DEBUG:teuthology.parallel:result is None 2022-04-02T03:17:03.118 INFO:teuthology.orchestra.run.smithi029.stdout:137438953484 2022-04-02T03:17:03.142 INFO:tasks.rook.ceph_manager.ceph:need seq 137438953484 got 137438953484 for osd.9 2022-04-02T03:17:03.143 DEBUG:teuthology.parallel:result is None 2022-04-02T03:17:03.238 INFO:teuthology.orchestra.run.smithi029.stdout:81604378643 2022-04-02T03:17:03.260 INFO:teuthology.orchestra.run.smithi029.stdout:98784247826 2022-04-02T03:17:03.263 INFO:tasks.rook.ceph_manager.ceph:need seq 81604378643 got 81604378643 for osd.2 2022-04-02T03:17:03.263 DEBUG:teuthology.parallel:result is None 2022-04-02T03:17:03.290 INFO:tasks.rook.ceph_manager.ceph:need seq 98784247826 got 98784247826 for osd.6 2022-04-02T03:17:03.290 DEBUG:teuthology.parallel:result is None 2022-04-02T03:17:03.347 INFO:teuthology.orchestra.run.smithi029.stdout:73014444052 2022-04-02T03:17:03.364 INFO:tasks.rook.ceph_manager.ceph:need seq 73014444052 got 73014444052 for osd.0 2022-04-02T03:17:03.365 DEBUG:teuthology.parallel:result is None 2022-04-02T03:17:03.381 INFO:teuthology.orchestra.run.smithi029.stdout:94489280529 2022-04-02T03:17:03.399 INFO:tasks.rook.ceph_manager.ceph:need seq 94489280529 got 94489280529 for osd.8 2022-04-02T03:17:03.399 DEBUG:teuthology.parallel:result is None 2022-04-02T03:17:03.406 INFO:teuthology.orchestra.run.smithi029.stdout:81604378643 2022-04-02T03:17:03.426 INFO:tasks.rook.ceph_manager.ceph:need seq 81604378643 got 81604378643 for osd.1 2022-04-02T03:17:03.427 DEBUG:teuthology.parallel:result is None 2022-04-02T03:17:03.427 INFO:tasks.rook.ceph_manager.ceph:waiting for clean 2022-04-02T03:17:03.428 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph pg dump --format=json 2022-04-02T03:17:03.983 INFO:teuthology.orchestra.run.smithi029.stdout: 2022-04-02T03:17:04.146 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2022-04-02T03:17:04.147 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":124,"stamp":"2022-04-02T03:17:02.848992+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":528928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":12,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":1124941824,"kb_used":67192,"kb_used_data":5976,"kb_used_omap":0,"kb_used_meta":61120,"kb_avail":1124874632,"statfs":{"total":1151940427776,"available":1151871623168,"internally_reserved":0,"allocated":6119424,"data_stored":3589668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":62586880},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004226"},"pg_stats":[{"pgid":"1.0","version":"24'117","reported_seq":203,"reported_epoch":36,"state":"active+clean","last_fresh":"2022-04-02T03:16:45.023107+0000","last_change":"2022-04-02T03:16:15.487085+0000","last_active":"2022-04-02T03:16:45.023107+0000","last_peered":"2022-04-02T03:16:45.023107+0000","last_clean":"2022-04-02T03:16:45.023107+0000","last_became_active":"2022-04-02T03:16:15.486262+0000","last_became_peered":"2022-04-02T03:16:15.486262+0000","last_unstale":"2022-04-02T03:16:45.023107+0000","last_undegraded":"2022-04-02T03:16:45.023107+0000","last_fullsized":"2022-04-02T03:16:45.023107+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-02T03:15:47.242453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-02T03:15:47.242453+0000","last_clean_scrub_stamp":"2022-04-02T03:15:47.242453+0000","objects_scrubbed":0,"log_size":117,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-03T06:12:21.631121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":528928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,3],"acting":[8,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":528928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2129920,"data_stored":2115712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":11,"up_from":36,"seq":154618822663,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5240,"kb_used_data":368,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93739912,"statfs":{"total":95995035648,"available":95989669888,"internally_reserved":0,"allocated":376832,"data_stored":166907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.032}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.645}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.878}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.548}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.569}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.198}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.827}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9390000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.496}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9790000000000001}]}]},{"osd":10,"up_from":32,"seq":137438953484,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5304,"kb_used_data":368,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739848,"statfs":{"total":95995035648,"available":95989604352,"internally_reserved":0,"allocated":376832,"data_stored":166907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1259999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1440000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7430000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4019999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79500000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0430000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.629}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.095}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.548}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1779999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.8769999999999998}]}]},{"osd":9,"up_from":32,"seq":137438953485,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5304,"kb_used_data":368,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739848,"statfs":{"total":95995035648,"available":95989604352,"internally_reserved":0,"allocated":376832,"data_stored":166907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.085}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0289999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.667}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.339}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.393}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.228}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1619999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.268}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.5110000000000001}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.518}]}]},{"osd":5,"up_from":28,"seq":120259084301,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5888,"kb_used_data":888,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":93739264,"statfs":{"total":95995035648,"available":95989006336,"internally_reserved":0,"allocated":909312,"data_stored":695835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.081}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3999999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.306}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58899999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7290000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3340000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.909}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4850000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.194}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.224}]}]},{"osd":7,"up_from":27,"seq":115964117006,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5304,"kb_used_data":368,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739848,"statfs":{"total":95995035648,"available":95989604352,"internally_reserved":0,"allocated":376832,"data_stored":166907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1080000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5660000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9950000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3149999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0619999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.786}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.391}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2629999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.319}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.016}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1909999999999998}]}]},{"osd":6,"up_from":23,"seq":98784247826,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5368,"kb_used_data":368,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":93739784,"statfs":{"total":95995035648,"available":95989538816,"internally_reserved":0,"allocated":376832,"data_stored":166907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 2 03:16:49 2022","interfaces":[{"interface":"back","average":{"1min":1.427,"5min":1.427,"15min":1.427},"min":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"max":{"1min":1.9490000000000001,"5min":1.9490000000000001,"15min":1.9490000000000001},"last":1.181},{"interface":"front","average":{"1min":1.22,"5min":1.22,"15min":1.22},"min":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"max":{"1min":1.8560000000000001,"5min":1.8560000000000001,"15min":1.8560000000000001},"last":1.645}]},{"osd":1,"last update":"Sat Apr 2 03:16:49 2022","interfaces":[{"interface":"back","average":{"1min":1.4159999999999999,"5min":1.4159999999999999,"15min":1.4159999999999999},"min":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"max":{"1min":2.294,"5min":2.294,"15min":2.294},"last":1.1339999999999999},{"interface":"front","average":{"1min":1.2390000000000001,"5min":1.2390000000000001,"15min":1.2390000000000001},"min":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"max":{"1min":1.734,"5min":1.734,"15min":1.734},"last":1.728}]},{"osd":2,"last update":"Sat Apr 2 03:16:49 2022","interfaces":[{"interface":"back","average":{"1min":1.3919999999999999,"5min":1.3919999999999999,"15min":1.3919999999999999},"min":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"max":{"1min":1.984,"5min":1.984,"15min":1.984},"last":0.97499999999999998},{"interface":"front","average":{"1min":1.5740000000000001,"5min":1.5740000000000001,"15min":1.5740000000000001},"min":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"max":{"1min":2.2240000000000002,"5min":2.2240000000000002,"15min":2.2240000000000002},"last":1.2470000000000001}]},{"osd":3,"last update":"Sat Apr 2 03:16:49 2022","interfaces":[{"interface":"back","average":{"1min":1.0529999999999999,"5min":1.0529999999999999,"15min":1.0529999999999999},"min":{"1min":0.626,"5min":0.626,"15min":0.626},"max":{"1min":1.952,"5min":1.952,"15min":1.952},"last":0.79800000000000004},{"interface":"front","average":{"1min":1.337,"5min":1.337,"15min":1.337},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":2.524,"5min":2.524,"15min":2.524},"last":0.82899999999999996}]},{"osd":4,"last update":"Sat Apr 2 03:16:49 2022","interfaces":[{"interface":"back","average":{"1min":1.389,"5min":1.389,"15min":1.389},"min":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":2.0529999999999999,"5min":2.0529999999999999,"15min":2.0529999999999999},"last":1.603},{"interface":"front","average":{"1min":1.2789999999999999,"5min":1.2789999999999999,"15min":1.2789999999999999},"min":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"max":{"1min":2.3300000000000001,"5min":2.3300000000000001,"15min":2.3300000000000001},"last":1.008}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90500000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.02}]},{"osd":8,"last update":"Sat Apr 2 03:16:49 2022","interfaces":[{"interface":"back","average":{"1min":1.3879999999999999,"5min":1.3879999999999999,"15min":1.3879999999999999},"min":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":1.853,"5min":1.853,"15min":1.853},"last":1.7789999999999999},{"interface":"front","average":{"1min":1.5900000000000001,"5min":1.5900000000000001,"15min":1.5900000000000001},"min":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"max":{"1min":2.2959999999999998,"5min":2.2959999999999998,"15min":2.2959999999999998},"last":6.9210000000000003}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.23}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6699999999999999}]}]},{"osd":1,"up_from":19,"seq":81604378643,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5368,"kb_used_data":368,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":93739784,"statfs":{"total":95995035648,"available":95989538816,"internally_reserved":0,"allocated":376832,"data_stored":166907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 2 03:16:44 2022","interfaces":[{"interface":"back","average":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"min":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"max":{"1min":1.349,"5min":1.349,"15min":1.349},"last":0.79900000000000004},{"interface":"front","average":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"min":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":1.49,"5min":1.49,"15min":1.49},"last":0.99199999999999999}]},{"osd":2,"last update":"Sat Apr 2 03:16:44 2022","interfaces":[{"interface":"back","average":{"1min":1.119,"5min":1.119,"15min":1.119},"min":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"max":{"1min":1.5229999999999999,"5min":1.5229999999999999,"15min":1.5229999999999999},"last":1.2749999999999999},{"interface":"front","average":{"1min":1.0609999999999999,"5min":1.0609999999999999,"15min":1.0609999999999999},"min":{"1min":0.75,"5min":0.75,"15min":0.75},"max":{"1min":1.341,"5min":1.341,"15min":1.341},"last":1.4390000000000001}]},{"osd":3,"last update":"Sat Apr 2 03:16:51 2022","interfaces":[{"interface":"back","average":{"1min":1.3009999999999999,"5min":1.3009999999999999,"15min":1.3009999999999999},"min":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"max":{"1min":2.0190000000000001,"5min":2.0190000000000001,"15min":2.0190000000000001},"last":1.597},{"interface":"front","average":{"1min":1.417,"5min":1.417,"15min":1.417},"min":{"1min":1.0940000000000001,"5min":1.0940000000000001,"15min":1.0940000000000001},"max":{"1min":2.2290000000000001,"5min":2.2290000000000001,"15min":2.2290000000000001},"last":1.1870000000000001}]},{"osd":4,"last update":"Sat Apr 2 03:16:45 2022","interfaces":[{"interface":"back","average":{"1min":1.367,"5min":1.367,"15min":1.367},"min":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"max":{"1min":2.1059999999999999,"5min":2.1059999999999999,"15min":2.1059999999999999},"last":1.4790000000000001},{"interface":"front","average":{"1min":1.4510000000000001,"5min":1.4510000000000001,"15min":1.4510000000000001},"min":{"1min":1,"5min":1,"15min":1},"max":{"1min":2.5390000000000001,"5min":2.5390000000000001,"15min":2.5390000000000001},"last":1.8120000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.516}]},{"osd":6,"last update":"Sat Apr 2 03:16:51 2022","interfaces":[{"interface":"back","average":{"1min":1.5509999999999999,"5min":1.5509999999999999,"15min":1.5509999999999999},"min":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"max":{"1min":2.6339999999999999,"5min":2.6339999999999999,"15min":2.6339999999999999},"last":1.99},{"interface":"front","average":{"1min":1.413,"5min":1.413,"15min":1.413},"min":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"max":{"1min":1.897,"5min":1.897,"15min":1.897},"last":1.5649999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2589999999999999}]},{"osd":8,"last update":"Sat Apr 2 03:16:51 2022","interfaces":[{"interface":"back","average":{"1min":1.7170000000000001,"5min":1.7170000000000001,"15min":1.7170000000000001},"min":{"1min":1.397,"5min":1.397,"15min":1.397},"max":{"1min":2.1880000000000002,"5min":2.1880000000000002,"15min":2.1880000000000002},"last":1.788},{"interface":"front","average":{"1min":1.8149999999999999,"5min":1.8149999999999999,"15min":1.8149999999999999},"min":{"1min":1.1539999999999999,"5min":1.1539999999999999,"15min":1.1539999999999999},"max":{"1min":3.1629999999999998,"5min":3.1629999999999998,"15min":3.1629999999999998},"last":1.9319999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.389}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.069}]}]},{"osd":8,"up_from":22,"seq":94489280529,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6400,"kb_used_data":888,"kb_used_omap":0,"kb_used_meta":5504,"kb_avail":93738752,"statfs":{"total":95995035648,"available":95988482048,"internally_reserved":0,"allocated":909312,"data_stored":695835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5636096},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 2 03:16:48 2022","interfaces":[{"interface":"back","average":{"1min":1.127,"5min":1.127,"15min":1.127},"min":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"max":{"1min":1.5249999999999999,"5min":1.5249999999999999,"15min":1.5249999999999999},"last":1.5780000000000001},{"interface":"front","average":{"1min":1.2669999999999999,"5min":1.2669999999999999,"15min":1.2669999999999999},"min":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"max":{"1min":1.7450000000000001,"5min":1.7450000000000001,"15min":1.7450000000000001},"last":1.601}]},{"osd":1,"last update":"Sat Apr 2 03:16:48 2022","interfaces":[{"interface":"back","average":{"1min":1.179,"5min":1.179,"15min":1.179},"min":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"max":{"1min":1.72,"5min":1.72,"15min":1.72},"last":1.54},{"interface":"front","average":{"1min":1.252,"5min":1.252,"15min":1.252},"min":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"max":{"1min":1.788,"5min":1.788,"15min":1.788},"last":1.706}]},{"osd":2,"last update":"Sat Apr 2 03:16:48 2022","interfaces":[{"interface":"back","average":{"1min":1.345,"5min":1.345,"15min":1.345},"min":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"max":{"1min":2.0920000000000001,"5min":2.0920000000000001,"15min":2.0920000000000001},"last":1.9890000000000001},{"interface":"front","average":{"1min":1.246,"5min":1.246,"15min":1.246},"min":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"max":{"1min":1.8939999999999999,"5min":1.8939999999999999,"15min":1.8939999999999999},"last":1.6639999999999999}]},{"osd":3,"last update":"Sat Apr 2 03:16:48 2022","interfaces":[{"interface":"back","average":{"1min":1.2130000000000001,"5min":1.2130000000000001,"15min":1.2130000000000001},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":2.0230000000000001,"5min":2.0230000000000001,"15min":2.0230000000000001},"last":1.054},{"interface":"front","average":{"1min":1.26,"5min":1.26,"15min":1.26},"min":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"max":{"1min":2.028,"5min":2.028,"15min":2.028},"last":1.333}]},{"osd":4,"last update":"Sat Apr 2 03:16:48 2022","interfaces":[{"interface":"back","average":{"1min":1.1619999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"min":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.8939999999999999,"5min":1.8939999999999999,"15min":1.8939999999999999},"last":1.3660000000000001},{"interface":"front","average":{"1min":1.385,"5min":1.385,"15min":1.385},"min":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"max":{"1min":1.9370000000000001,"5min":1.9370000000000001,"15min":1.9370000000000001},"last":1.671}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8500000000000001}]},{"osd":6,"last update":"Sat Apr 2 03:16:51 2022","interfaces":[{"interface":"back","average":{"1min":1.349,"5min":1.349,"15min":1.349},"min":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"max":{"1min":2.121,"5min":2.121,"15min":2.121},"last":1.466},{"interface":"front","average":{"1min":1.405,"5min":1.405,"15min":1.405},"min":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"max":{"1min":2.0009999999999999,"5min":2.0009999999999999,"15min":2.0009999999999999},"last":1.49}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.77}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9490000000000001}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8220000000000001}]}]},{"osd":0,"up_from":17,"seq":73014444052,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5880,"kb_used_data":368,"kb_used_omap":0,"kb_used_meta":5504,"kb_avail":93739272,"statfs":{"total":95995035648,"available":95989014528,"internally_reserved":0,"allocated":376832,"data_stored":166907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5636096},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 2 03:16:43 2022","interfaces":[{"interface":"back","average":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.331,"5min":1.331,"15min":1.331},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.337,"5min":1.337,"15min":1.337},"last":1.3779999999999999}]},{"osd":2,"last update":"Sat Apr 2 03:16:43 2022","interfaces":[{"interface":"back","average":{"1min":1.036,"5min":1.036,"15min":1.036},"min":{"1min":0.626,"5min":0.626,"15min":0.626},"max":{"1min":2.0129999999999999,"5min":2.0129999999999999,"15min":2.0129999999999999},"last":1.131},{"interface":"front","average":{"1min":1.0660000000000001,"5min":1.0660000000000001,"15min":1.0660000000000001},"min":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"max":{"1min":1.4099999999999999,"5min":1.4099999999999999,"15min":1.4099999999999999},"last":1.169}]},{"osd":3,"last update":"Sat Apr 2 03:16:48 2022","interfaces":[{"interface":"back","average":{"1min":1.335,"5min":1.335,"15min":1.335},"min":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.821,"5min":1.821,"15min":1.821},"last":1.4890000000000001},{"interface":"front","average":{"1min":1.3380000000000001,"5min":1.3380000000000001,"15min":1.3380000000000001},"min":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"max":{"1min":1.9390000000000001,"5min":1.9390000000000001,"15min":1.9390000000000001},"last":1.01}]},{"osd":4,"last update":"Sat Apr 2 03:16:45 2022","interfaces":[{"interface":"back","average":{"1min":1.3879999999999999,"5min":1.3879999999999999,"15min":1.3879999999999999},"min":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"max":{"1min":2.3010000000000002,"5min":2.3010000000000002,"15min":2.3010000000000002},"last":1.7430000000000001},{"interface":"front","average":{"1min":1.5029999999999999,"5min":1.5029999999999999,"15min":1.5029999999999999},"min":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"max":{"1min":2.149,"5min":2.149,"15min":2.149},"last":1.3069999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96099999999999997}]},{"osd":6,"last update":"Sat Apr 2 03:16:52 2022","interfaces":[{"interface":"back","average":{"1min":1.542,"5min":1.542,"15min":1.542},"min":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"max":{"1min":2.4430000000000001,"5min":2.4430000000000001,"15min":2.4430000000000001},"last":1.5129999999999999},{"interface":"front","average":{"1min":1.637,"5min":1.637,"15min":1.637},"min":{"1min":1.008,"5min":1.008,"15min":1.008},"max":{"1min":2.2410000000000001,"5min":2.2410000000000001,"15min":2.2410000000000001},"last":1.0760000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7609999999999999}]},{"osd":8,"last update":"Sat Apr 2 03:16:48 2022","interfaces":[{"interface":"back","average":{"1min":1.7729999999999999,"5min":1.7729999999999999,"15min":1.7729999999999999},"min":{"1min":1.238,"5min":1.238,"15min":1.238},"max":{"1min":2.5049999999999999,"5min":2.5049999999999999,"15min":2.5049999999999999},"last":1.552},{"interface":"front","average":{"1min":1.7410000000000001,"5min":1.7410000000000001,"15min":1.7410000000000001},"min":{"1min":1.27,"5min":1.27,"15min":1.27},"max":{"1min":2.4369999999999998,"5min":2.4369999999999998,"15min":2.4369999999999998},"last":1.6799999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6259999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.798}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.28}]}]},{"osd":2,"up_from":19,"seq":81604378643,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5368,"kb_used_data":368,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":93739784,"statfs":{"total":95995035648,"available":95989538816,"internally_reserved":0,"allocated":376832,"data_stored":166907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 2 03:16:46 2022","interfaces":[{"interface":"back","average":{"1min":1.0089999999999999,"5min":1.0089999999999999,"15min":1.0089999999999999},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":2.0750000000000002,"5min":2.0750000000000002,"15min":2.0750000000000002},"last":0.90500000000000003},{"interface":"front","average":{"1min":1.1779999999999999,"5min":1.1779999999999999,"15min":1.1779999999999999},"min":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"max":{"1min":1.671,"5min":1.671,"15min":1.671},"last":1.052}]},{"osd":1,"last update":"Sat Apr 2 03:16:46 2022","interfaces":[{"interface":"back","average":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"min":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"max":{"1min":1.5680000000000001,"5min":1.5680000000000001,"15min":1.5680000000000001},"last":0.998},{"interface":"front","average":{"1min":1.0409999999999999,"5min":1.0409999999999999,"15min":1.0409999999999999},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.6739999999999999,"5min":1.6739999999999999,"15min":1.6739999999999999},"last":0.95599999999999996}]},{"osd":3,"last update":"Sat Apr 2 03:16:54 2022","interfaces":[{"interface":"back","average":{"1min":1.3220000000000001,"5min":1.3220000000000001,"15min":1.3220000000000001},"min":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"max":{"1min":2.0430000000000001,"5min":2.0430000000000001,"15min":2.0430000000000001},"last":1.264},{"interface":"front","average":{"1min":1.411,"5min":1.411,"15min":1.411},"min":{"1min":1.02,"5min":1.02,"15min":1.02},"max":{"1min":2.23,"5min":2.23,"15min":2.23},"last":1.387}]},{"osd":4,"last update":"Sat Apr 2 03:16:46 2022","interfaces":[{"interface":"back","average":{"1min":1.4259999999999999,"5min":1.4259999999999999,"15min":1.4259999999999999},"min":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"max":{"1min":1.8420000000000001,"5min":1.8420000000000001,"15min":1.8420000000000001},"last":1.425},{"interface":"front","average":{"1min":1.4410000000000001,"5min":1.4410000000000001,"15min":1.4410000000000001},"min":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"max":{"1min":2.1819999999999999,"5min":2.1819999999999999,"15min":2.1819999999999999},"last":1.306}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.73}]},{"osd":6,"last update":"Sat Apr 2 03:16:54 2022","interfaces":[{"interface":"back","average":{"1min":1.5840000000000001,"5min":1.5840000000000001,"15min":1.5840000000000001},"min":{"1min":1.1499999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"max":{"1min":2.0939999999999999,"5min":2.0939999999999999,"15min":2.0939999999999999},"last":1.5860000000000001},{"interface":"front","average":{"1min":1.615,"5min":1.615,"15min":1.615},"min":{"1min":1.137,"5min":1.137,"15min":1.137},"max":{"1min":2.254,"5min":2.254,"15min":2.254},"last":1.361}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.988}]},{"osd":8,"last update":"Sat Apr 2 03:16:54 2022","interfaces":[{"interface":"back","average":{"1min":1.7150000000000001,"5min":1.7150000000000001,"15min":1.7150000000000001},"min":{"1min":1.03,"5min":1.03,"15min":1.03},"max":{"1min":2.5390000000000001,"5min":2.5390000000000001,"15min":2.5390000000000001},"last":1.6619999999999999},{"interface":"front","average":{"1min":1.71,"5min":1.71,"15min":1.71},"min":{"1min":1.395,"5min":1.395,"15min":1.395},"max":{"1min":2.0609999999999999,"5min":2.0609999999999999,"15min":2.0609999999999999},"last":1.651}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8560000000000001}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.903}]}]},{"osd":4,"up_from":20,"seq":85899345942,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5368,"kb_used_data":368,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":93739784,"statfs":{"total":95995035648,"available":95989538816,"internally_reserved":0,"allocated":376832,"data_stored":166907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 2 03:16:46 2022","interfaces":[{"interface":"back","average":{"1min":1.2989999999999999,"5min":1.2989999999999999,"15min":1.2989999999999999},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":2.056,"5min":2.056,"15min":2.056},"last":1.3899999999999999},{"interface":"front","average":{"1min":1.202,"5min":1.202,"15min":1.202},"min":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"max":{"1min":1.9079999999999999,"5min":1.9079999999999999,"15min":1.9079999999999999},"last":1.514}]},{"osd":1,"last update":"Sat Apr 2 03:16:46 2022","interfaces":[{"interface":"back","average":{"1min":1.395,"5min":1.395,"15min":1.395},"min":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"max":{"1min":1.9359999999999999,"5min":1.9359999999999999,"15min":1.9359999999999999},"last":2.2389999999999999},{"interface":"front","average":{"1min":1.4259999999999999,"5min":1.4259999999999999,"15min":1.4259999999999999},"min":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"max":{"1min":2.1099999999999999,"5min":2.1099999999999999,"15min":2.1099999999999999},"last":1.8009999999999999}]},{"osd":2,"last update":"Sat Apr 2 03:16:46 2022","interfaces":[{"interface":"back","average":{"1min":1.2749999999999999,"5min":1.2749999999999999,"15min":1.2749999999999999},"min":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"max":{"1min":1.9019999999999999,"5min":1.9019999999999999,"15min":1.9019999999999999},"last":1.552},{"interface":"front","average":{"1min":1.48,"5min":1.48,"15min":1.48},"min":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"max":{"1min":2.2349999999999999,"5min":2.2349999999999999,"15min":2.2349999999999999},"last":1.452}]},{"osd":3,"last update":"Sat Apr 2 03:16:53 2022","interfaces":[{"interface":"back","average":{"1min":1.1080000000000001,"5min":1.1080000000000001,"15min":1.1080000000000001},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.7430000000000001,"5min":1.7430000000000001,"15min":1.7430000000000001},"last":1.6910000000000001},{"interface":"front","average":{"1min":1.2669999999999999,"5min":1.2669999999999999,"15min":1.2669999999999999},"min":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"max":{"1min":1.9139999999999999,"5min":1.9139999999999999,"15min":1.9139999999999999},"last":1.319}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8640000000000001}]},{"osd":6,"last update":"Sat Apr 2 03:16:53 2022","interfaces":[{"interface":"back","average":{"1min":1.5920000000000001,"5min":1.5920000000000001,"15min":1.5920000000000001},"min":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":2.3170000000000002,"5min":2.3170000000000002,"15min":2.3170000000000002},"last":2.706},{"interface":"front","average":{"1min":1.5609999999999999,"5min":1.5609999999999999,"15min":1.5609999999999999},"min":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":2.7989999999999999,"5min":2.7989999999999999,"15min":2.7989999999999999},"last":1.986}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.8140000000000001}]},{"osd":8,"last update":"Sat Apr 2 03:16:53 2022","interfaces":[{"interface":"back","average":{"1min":1.621,"5min":1.621,"15min":1.621},"min":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"max":{"1min":2.375,"5min":2.375,"15min":2.375},"last":1.669},{"interface":"front","average":{"1min":1.831,"5min":1.831,"15min":1.831},"min":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"max":{"1min":3.1389999999999998,"5min":3.1389999999999998,"15min":3.1389999999999998},"last":2.964}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.8980000000000001}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.339}]}]},{"osd":3,"up_from":22,"seq":94489280530,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6400,"kb_used_data":888,"kb_used_omap":0,"kb_used_meta":5504,"kb_avail":93738752,"statfs":{"total":95995035648,"available":95988482048,"internally_reserved":0,"allocated":909312,"data_stored":695835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5636096},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 2 03:16:48 2022","interfaces":[{"interface":"back","average":{"1min":1.3200000000000001,"5min":1.3200000000000001,"15min":1.3200000000000001},"min":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":2.1349999999999998,"5min":2.1349999999999998,"15min":2.1349999999999998},"last":1.5589999999999999},{"interface":"front","average":{"1min":1.3540000000000001,"5min":1.3540000000000001,"15min":1.3540000000000001},"min":{"1min":0.747,"5min":0.747,"15min":0.747},"max":{"1min":2.2629999999999999,"5min":2.2629999999999999,"15min":2.2629999999999999},"last":1.248}]},{"osd":1,"last update":"Sat Apr 2 03:16:48 2022","interfaces":[{"interface":"back","average":{"1min":1.335,"5min":1.335,"15min":1.335},"min":{"1min":1,"5min":1,"15min":1},"max":{"1min":2.258,"5min":2.258,"15min":2.258},"last":1.417},{"interface":"front","average":{"1min":1.3280000000000001,"5min":1.3280000000000001,"15min":1.3280000000000001},"min":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"max":{"1min":3.1869999999999998,"5min":3.1869999999999998,"15min":3.1869999999999998},"last":1.353}]},{"osd":2,"last update":"Sat Apr 2 03:16:48 2022","interfaces":[{"interface":"back","average":{"1min":1.4530000000000001,"5min":1.4530000000000001,"15min":1.4530000000000001},"min":{"1min":1.143,"5min":1.143,"15min":1.143},"max":{"1min":2.3860000000000001,"5min":2.3860000000000001,"15min":2.3860000000000001},"last":1.677},{"interface":"front","average":{"1min":1.621,"5min":1.621,"15min":1.621},"min":{"1min":1.141,"5min":1.141,"15min":1.141},"max":{"1min":5.0439999999999996,"5min":5.0439999999999996,"15min":5.0439999999999996},"last":1.853}]},{"osd":4,"last update":"Sat Apr 2 03:16:48 2022","interfaces":[{"interface":"back","average":{"1min":1.2849999999999999,"5min":1.2849999999999999,"15min":1.2849999999999999},"min":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":3.0800000000000001,"5min":3.0800000000000001,"15min":3.0800000000000001},"last":1.774},{"interface":"front","average":{"1min":1.254,"5min":1.254,"15min":1.254},"min":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"max":{"1min":1.8500000000000001,"5min":1.8500000000000001,"15min":1.8500000000000001},"last":2.0539999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5900000000000001}]},{"osd":6,"last update":"Sat Apr 2 03:16:51 2022","interfaces":[{"interface":"back","average":{"1min":1.446,"5min":1.446,"15min":1.446},"min":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"max":{"1min":3.1389999999999998,"5min":3.1389999999999998,"15min":3.1389999999999998},"last":1.0589999999999999},{"interface":"front","average":{"1min":1.581,"5min":1.581,"15min":1.581},"min":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"max":{"1min":2.4020000000000001,"5min":2.4020000000000001,"15min":2.4020000000000001},"last":1.7430000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8859999999999999}]},{"osd":8,"last update":"Sat Apr 2 03:16:48 2022","interfaces":[{"interface":"back","average":{"1min":1.849,"5min":1.849,"15min":1.849},"min":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"max":{"1min":5.5519999999999996,"5min":5.5519999999999996,"15min":5.5519999999999996},"last":1.724},{"interface":"front","average":{"1min":1.681,"5min":1.681,"15min":1.681},"min":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"max":{"1min":3.3919999999999999,"5min":3.3919999999999999,"15min":3.3919999999999999},"last":1.9199999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.387}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.536}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2022-04-02T03:17:04.150 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph pg dump --format=json 2022-04-02T03:17:04.647 INFO:teuthology.orchestra.run.smithi029.stdout: 2022-04-02T03:17:04.649 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2022-04-02T03:17:04.666 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":124,"stamp":"2022-04-02T03:17:02.848992+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":528928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":12,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":1124941824,"kb_used":67192,"kb_used_data":5976,"kb_used_omap":0,"kb_used_meta":61120,"kb_avail":1124874632,"statfs":{"total":1151940427776,"available":1151871623168,"internally_reserved":0,"allocated":6119424,"data_stored":3589668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":62586880},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004226"},"pg_stats":[{"pgid":"1.0","version":"24'117","reported_seq":203,"reported_epoch":36,"state":"active+clean","last_fresh":"2022-04-02T03:16:45.023107+0000","last_change":"2022-04-02T03:16:15.487085+0000","last_active":"2022-04-02T03:16:45.023107+0000","last_peered":"2022-04-02T03:16:45.023107+0000","last_clean":"2022-04-02T03:16:45.023107+0000","last_became_active":"2022-04-02T03:16:15.486262+0000","last_became_peered":"2022-04-02T03:16:15.486262+0000","last_unstale":"2022-04-02T03:16:45.023107+0000","last_undegraded":"2022-04-02T03:16:45.023107+0000","last_fullsized":"2022-04-02T03:16:45.023107+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-02T03:15:47.242453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-02T03:15:47.242453+0000","last_clean_scrub_stamp":"2022-04-02T03:15:47.242453+0000","objects_scrubbed":0,"log_size":117,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-03T06:12:21.631121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":528928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,3],"acting":[8,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":528928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2129920,"data_stored":2115712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":11,"up_from":36,"seq":154618822663,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5240,"kb_used_data":368,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93739912,"statfs":{"total":95995035648,"available":95989669888,"internally_reserved":0,"allocated":376832,"data_stored":166907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.032}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.645}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.878}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.548}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.569}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.198}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.827}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9390000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.496}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9790000000000001}]}]},{"osd":10,"up_from":32,"seq":137438953484,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5304,"kb_used_data":368,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739848,"statfs":{"total":95995035648,"available":95989604352,"internally_reserved":0,"allocated":376832,"data_stored":166907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1259999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1440000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7430000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4019999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79500000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0430000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.629}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.095}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.548}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1779999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.8769999999999998}]}]},{"osd":9,"up_from":32,"seq":137438953485,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5304,"kb_used_data":368,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739848,"statfs":{"total":95995035648,"available":95989604352,"internally_reserved":0,"allocated":376832,"data_stored":166907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.085}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0289999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.667}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.339}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.393}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.228}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1619999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.268}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.5110000000000001}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.518}]}]},{"osd":5,"up_from":28,"seq":120259084301,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5888,"kb_used_data":888,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":93739264,"statfs":{"total":95995035648,"available":95989006336,"internally_reserved":0,"allocated":909312,"data_stored":695835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.081}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3999999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.306}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58899999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7290000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3340000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.909}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4850000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.194}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.224}]}]},{"osd":7,"up_from":27,"seq":115964117006,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5304,"kb_used_data":368,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739848,"statfs":{"total":95995035648,"available":95989604352,"internally_reserved":0,"allocated":376832,"data_stored":166907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1080000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5660000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9950000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3149999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0619999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.786}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.391}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2629999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.319}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.016}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1909999999999998}]}]},{"osd":6,"up_from":23,"seq":98784247826,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5368,"kb_used_data":368,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":93739784,"statfs":{"total":95995035648,"available":95989538816,"internally_reserved":0,"allocated":376832,"data_stored":166907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 2 03:16:49 2022","interfaces":[{"interface":"back","average":{"1min":1.427,"5min":1.427,"15min":1.427},"min":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"max":{"1min":1.9490000000000001,"5min":1.9490000000000001,"15min":1.9490000000000001},"last":1.181},{"interface":"front","average":{"1min":1.22,"5min":1.22,"15min":1.22},"min":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"max":{"1min":1.8560000000000001,"5min":1.8560000000000001,"15min":1.8560000000000001},"last":1.645}]},{"osd":1,"last update":"Sat Apr 2 03:16:49 2022","interfaces":[{"interface":"back","average":{"1min":1.4159999999999999,"5min":1.4159999999999999,"15min":1.4159999999999999},"min":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"max":{"1min":2.294,"5min":2.294,"15min":2.294},"last":1.1339999999999999},{"interface":"front","average":{"1min":1.2390000000000001,"5min":1.2390000000000001,"15min":1.2390000000000001},"min":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"max":{"1min":1.734,"5min":1.734,"15min":1.734},"last":1.728}]},{"osd":2,"last update":"Sat Apr 2 03:16:49 2022","interfaces":[{"interface":"back","average":{"1min":1.3919999999999999,"5min":1.3919999999999999,"15min":1.3919999999999999},"min":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"max":{"1min":1.984,"5min":1.984,"15min":1.984},"last":0.97499999999999998},{"interface":"front","average":{"1min":1.5740000000000001,"5min":1.5740000000000001,"15min":1.5740000000000001},"min":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"max":{"1min":2.2240000000000002,"5min":2.2240000000000002,"15min":2.2240000000000002},"last":1.2470000000000001}]},{"osd":3,"last update":"Sat Apr 2 03:16:49 2022","interfaces":[{"interface":"back","average":{"1min":1.0529999999999999,"5min":1.0529999999999999,"15min":1.0529999999999999},"min":{"1min":0.626,"5min":0.626,"15min":0.626},"max":{"1min":1.952,"5min":1.952,"15min":1.952},"last":0.79800000000000004},{"interface":"front","average":{"1min":1.337,"5min":1.337,"15min":1.337},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":2.524,"5min":2.524,"15min":2.524},"last":0.82899999999999996}]},{"osd":4,"last update":"Sat Apr 2 03:16:49 2022","interfaces":[{"interface":"back","average":{"1min":1.389,"5min":1.389,"15min":1.389},"min":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":2.0529999999999999,"5min":2.0529999999999999,"15min":2.0529999999999999},"last":1.603},{"interface":"front","average":{"1min":1.2789999999999999,"5min":1.2789999999999999,"15min":1.2789999999999999},"min":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"max":{"1min":2.3300000000000001,"5min":2.3300000000000001,"15min":2.3300000000000001},"last":1.008}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90500000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.02}]},{"osd":8,"last update":"Sat Apr 2 03:16:49 2022","interfaces":[{"interface":"back","average":{"1min":1.3879999999999999,"5min":1.3879999999999999,"15min":1.3879999999999999},"min":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":1.853,"5min":1.853,"15min":1.853},"last":1.7789999999999999},{"interface":"front","average":{"1min":1.5900000000000001,"5min":1.5900000000000001,"15min":1.5900000000000001},"min":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"max":{"1min":2.2959999999999998,"5min":2.2959999999999998,"15min":2.2959999999999998},"last":6.9210000000000003}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.23}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6699999999999999}]}]},{"osd":1,"up_from":19,"seq":81604378643,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5368,"kb_used_data":368,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":93739784,"statfs":{"total":95995035648,"available":95989538816,"internally_reserved":0,"allocated":376832,"data_stored":166907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 2 03:16:44 2022","interfaces":[{"interface":"back","average":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"min":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"max":{"1min":1.349,"5min":1.349,"15min":1.349},"last":0.79900000000000004},{"interface":"front","average":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"min":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":1.49,"5min":1.49,"15min":1.49},"last":0.99199999999999999}]},{"osd":2,"last update":"Sat Apr 2 03:16:44 2022","interfaces":[{"interface":"back","average":{"1min":1.119,"5min":1.119,"15min":1.119},"min":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"max":{"1min":1.5229999999999999,"5min":1.5229999999999999,"15min":1.5229999999999999},"last":1.2749999999999999},{"interface":"front","average":{"1min":1.0609999999999999,"5min":1.0609999999999999,"15min":1.0609999999999999},"min":{"1min":0.75,"5min":0.75,"15min":0.75},"max":{"1min":1.341,"5min":1.341,"15min":1.341},"last":1.4390000000000001}]},{"osd":3,"last update":"Sat Apr 2 03:16:51 2022","interfaces":[{"interface":"back","average":{"1min":1.3009999999999999,"5min":1.3009999999999999,"15min":1.3009999999999999},"min":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"max":{"1min":2.0190000000000001,"5min":2.0190000000000001,"15min":2.0190000000000001},"last":1.597},{"interface":"front","average":{"1min":1.417,"5min":1.417,"15min":1.417},"min":{"1min":1.0940000000000001,"5min":1.0940000000000001,"15min":1.0940000000000001},"max":{"1min":2.2290000000000001,"5min":2.2290000000000001,"15min":2.2290000000000001},"last":1.1870000000000001}]},{"osd":4,"last update":"Sat Apr 2 03:16:45 2022","interfaces":[{"interface":"back","average":{"1min":1.367,"5min":1.367,"15min":1.367},"min":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"max":{"1min":2.1059999999999999,"5min":2.1059999999999999,"15min":2.1059999999999999},"last":1.4790000000000001},{"interface":"front","average":{"1min":1.4510000000000001,"5min":1.4510000000000001,"15min":1.4510000000000001},"min":{"1min":1,"5min":1,"15min":1},"max":{"1min":2.5390000000000001,"5min":2.5390000000000001,"15min":2.5390000000000001},"last":1.8120000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.516}]},{"osd":6,"last update":"Sat Apr 2 03:16:51 2022","interfaces":[{"interface":"back","average":{"1min":1.5509999999999999,"5min":1.5509999999999999,"15min":1.5509999999999999},"min":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"max":{"1min":2.6339999999999999,"5min":2.6339999999999999,"15min":2.6339999999999999},"last":1.99},{"interface":"front","average":{"1min":1.413,"5min":1.413,"15min":1.413},"min":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"max":{"1min":1.897,"5min":1.897,"15min":1.897},"last":1.5649999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2589999999999999}]},{"osd":8,"last update":"Sat Apr 2 03:16:51 2022","interfaces":[{"interface":"back","average":{"1min":1.7170000000000001,"5min":1.7170000000000001,"15min":1.7170000000000001},"min":{"1min":1.397,"5min":1.397,"15min":1.397},"max":{"1min":2.1880000000000002,"5min":2.1880000000000002,"15min":2.1880000000000002},"last":1.788},{"interface":"front","average":{"1min":1.8149999999999999,"5min":1.8149999999999999,"15min":1.8149999999999999},"min":{"1min":1.1539999999999999,"5min":1.1539999999999999,"15min":1.1539999999999999},"max":{"1min":3.1629999999999998,"5min":3.1629999999999998,"15min":3.1629999999999998},"last":1.9319999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.389}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.069}]}]},{"osd":8,"up_from":22,"seq":94489280529,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6400,"kb_used_data":888,"kb_used_omap":0,"kb_used_meta":5504,"kb_avail":93738752,"statfs":{"total":95995035648,"available":95988482048,"internally_reserved":0,"allocated":909312,"data_stored":695835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5636096},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 2 03:16:48 2022","interfaces":[{"interface":"back","average":{"1min":1.127,"5min":1.127,"15min":1.127},"min":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"max":{"1min":1.5249999999999999,"5min":1.5249999999999999,"15min":1.5249999999999999},"last":1.5780000000000001},{"interface":"front","average":{"1min":1.2669999999999999,"5min":1.2669999999999999,"15min":1.2669999999999999},"min":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"max":{"1min":1.7450000000000001,"5min":1.7450000000000001,"15min":1.7450000000000001},"last":1.601}]},{"osd":1,"last update":"Sat Apr 2 03:16:48 2022","interfaces":[{"interface":"back","average":{"1min":1.179,"5min":1.179,"15min":1.179},"min":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"max":{"1min":1.72,"5min":1.72,"15min":1.72},"last":1.54},{"interface":"front","average":{"1min":1.252,"5min":1.252,"15min":1.252},"min":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"max":{"1min":1.788,"5min":1.788,"15min":1.788},"last":1.706}]},{"osd":2,"last update":"Sat Apr 2 03:16:48 2022","interfaces":[{"interface":"back","average":{"1min":1.345,"5min":1.345,"15min":1.345},"min":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"max":{"1min":2.0920000000000001,"5min":2.0920000000000001,"15min":2.0920000000000001},"last":1.9890000000000001},{"interface":"front","average":{"1min":1.246,"5min":1.246,"15min":1.246},"min":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"max":{"1min":1.8939999999999999,"5min":1.8939999999999999,"15min":1.8939999999999999},"last":1.6639999999999999}]},{"osd":3,"last update":"Sat Apr 2 03:16:48 2022","interfaces":[{"interface":"back","average":{"1min":1.2130000000000001,"5min":1.2130000000000001,"15min":1.2130000000000001},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":2.0230000000000001,"5min":2.0230000000000001,"15min":2.0230000000000001},"last":1.054},{"interface":"front","average":{"1min":1.26,"5min":1.26,"15min":1.26},"min":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"max":{"1min":2.028,"5min":2.028,"15min":2.028},"last":1.333}]},{"osd":4,"last update":"Sat Apr 2 03:16:48 2022","interfaces":[{"interface":"back","average":{"1min":1.1619999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"min":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.8939999999999999,"5min":1.8939999999999999,"15min":1.8939999999999999},"last":1.3660000000000001},{"interface":"front","average":{"1min":1.385,"5min":1.385,"15min":1.385},"min":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"max":{"1min":1.9370000000000001,"5min":1.9370000000000001,"15min":1.9370000000000001},"last":1.671}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8500000000000001}]},{"osd":6,"last update":"Sat Apr 2 03:16:51 2022","interfaces":[{"interface":"back","average":{"1min":1.349,"5min":1.349,"15min":1.349},"min":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"max":{"1min":2.121,"5min":2.121,"15min":2.121},"last":1.466},{"interface":"front","average":{"1min":1.405,"5min":1.405,"15min":1.405},"min":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"max":{"1min":2.0009999999999999,"5min":2.0009999999999999,"15min":2.0009999999999999},"last":1.49}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.77}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9490000000000001}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8220000000000001}]}]},{"osd":0,"up_from":17,"seq":73014444052,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5880,"kb_used_data":368,"kb_used_omap":0,"kb_used_meta":5504,"kb_avail":93739272,"statfs":{"total":95995035648,"available":95989014528,"internally_reserved":0,"allocated":376832,"data_stored":166907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5636096},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 2 03:16:43 2022","interfaces":[{"interface":"back","average":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.331,"5min":1.331,"15min":1.331},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.337,"5min":1.337,"15min":1.337},"last":1.3779999999999999}]},{"osd":2,"last update":"Sat Apr 2 03:16:43 2022","interfaces":[{"interface":"back","average":{"1min":1.036,"5min":1.036,"15min":1.036},"min":{"1min":0.626,"5min":0.626,"15min":0.626},"max":{"1min":2.0129999999999999,"5min":2.0129999999999999,"15min":2.0129999999999999},"last":1.131},{"interface":"front","average":{"1min":1.0660000000000001,"5min":1.0660000000000001,"15min":1.0660000000000001},"min":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"max":{"1min":1.4099999999999999,"5min":1.4099999999999999,"15min":1.4099999999999999},"last":1.169}]},{"osd":3,"last update":"Sat Apr 2 03:16:48 2022","interfaces":[{"interface":"back","average":{"1min":1.335,"5min":1.335,"15min":1.335},"min":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.821,"5min":1.821,"15min":1.821},"last":1.4890000000000001},{"interface":"front","average":{"1min":1.3380000000000001,"5min":1.3380000000000001,"15min":1.3380000000000001},"min":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"max":{"1min":1.9390000000000001,"5min":1.9390000000000001,"15min":1.9390000000000001},"last":1.01}]},{"osd":4,"last update":"Sat Apr 2 03:16:45 2022","interfaces":[{"interface":"back","average":{"1min":1.3879999999999999,"5min":1.3879999999999999,"15min":1.3879999999999999},"min":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"max":{"1min":2.3010000000000002,"5min":2.3010000000000002,"15min":2.3010000000000002},"last":1.7430000000000001},{"interface":"front","average":{"1min":1.5029999999999999,"5min":1.5029999999999999,"15min":1.5029999999999999},"min":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"max":{"1min":2.149,"5min":2.149,"15min":2.149},"last":1.3069999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96099999999999997}]},{"osd":6,"last update":"Sat Apr 2 03:16:52 2022","interfaces":[{"interface":"back","average":{"1min":1.542,"5min":1.542,"15min":1.542},"min":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"max":{"1min":2.4430000000000001,"5min":2.4430000000000001,"15min":2.4430000000000001},"last":1.5129999999999999},{"interface":"front","average":{"1min":1.637,"5min":1.637,"15min":1.637},"min":{"1min":1.008,"5min":1.008,"15min":1.008},"max":{"1min":2.2410000000000001,"5min":2.2410000000000001,"15min":2.2410000000000001},"last":1.0760000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7609999999999999}]},{"osd":8,"last update":"Sat Apr 2 03:16:48 2022","interfaces":[{"interface":"back","average":{"1min":1.7729999999999999,"5min":1.7729999999999999,"15min":1.7729999999999999},"min":{"1min":1.238,"5min":1.238,"15min":1.238},"max":{"1min":2.5049999999999999,"5min":2.5049999999999999,"15min":2.5049999999999999},"last":1.552},{"interface":"front","average":{"1min":1.7410000000000001,"5min":1.7410000000000001,"15min":1.7410000000000001},"min":{"1min":1.27,"5min":1.27,"15min":1.27},"max":{"1min":2.4369999999999998,"5min":2.4369999999999998,"15min":2.4369999999999998},"last":1.6799999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6259999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.798}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.28}]}]},{"osd":2,"up_from":19,"seq":81604378643,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5368,"kb_used_data":368,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":93739784,"statfs":{"total":95995035648,"available":95989538816,"internally_reserved":0,"allocated":376832,"data_stored":166907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 2 03:16:46 2022","interfaces":[{"interface":"back","average":{"1min":1.0089999999999999,"5min":1.0089999999999999,"15min":1.0089999999999999},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":2.0750000000000002,"5min":2.0750000000000002,"15min":2.0750000000000002},"last":0.90500000000000003},{"interface":"front","average":{"1min":1.1779999999999999,"5min":1.1779999999999999,"15min":1.1779999999999999},"min":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"max":{"1min":1.671,"5min":1.671,"15min":1.671},"last":1.052}]},{"osd":1,"last update":"Sat Apr 2 03:16:46 2022","interfaces":[{"interface":"back","average":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"min":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"max":{"1min":1.5680000000000001,"5min":1.5680000000000001,"15min":1.5680000000000001},"last":0.998},{"interface":"front","average":{"1min":1.0409999999999999,"5min":1.0409999999999999,"15min":1.0409999999999999},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.6739999999999999,"5min":1.6739999999999999,"15min":1.6739999999999999},"last":0.95599999999999996}]},{"osd":3,"last update":"Sat Apr 2 03:16:54 2022","interfaces":[{"interface":"back","average":{"1min":1.3220000000000001,"5min":1.3220000000000001,"15min":1.3220000000000001},"min":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"max":{"1min":2.0430000000000001,"5min":2.0430000000000001,"15min":2.0430000000000001},"last":1.264},{"interface":"front","average":{"1min":1.411,"5min":1.411,"15min":1.411},"min":{"1min":1.02,"5min":1.02,"15min":1.02},"max":{"1min":2.23,"5min":2.23,"15min":2.23},"last":1.387}]},{"osd":4,"last update":"Sat Apr 2 03:16:46 2022","interfaces":[{"interface":"back","average":{"1min":1.4259999999999999,"5min":1.4259999999999999,"15min":1.4259999999999999},"min":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"max":{"1min":1.8420000000000001,"5min":1.8420000000000001,"15min":1.8420000000000001},"last":1.425},{"interface":"front","average":{"1min":1.4410000000000001,"5min":1.4410000000000001,"15min":1.4410000000000001},"min":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"max":{"1min":2.1819999999999999,"5min":2.1819999999999999,"15min":2.1819999999999999},"last":1.306}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.73}]},{"osd":6,"last update":"Sat Apr 2 03:16:54 2022","interfaces":[{"interface":"back","average":{"1min":1.5840000000000001,"5min":1.5840000000000001,"15min":1.5840000000000001},"min":{"1min":1.1499999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"max":{"1min":2.0939999999999999,"5min":2.0939999999999999,"15min":2.0939999999999999},"last":1.5860000000000001},{"interface":"front","average":{"1min":1.615,"5min":1.615,"15min":1.615},"min":{"1min":1.137,"5min":1.137,"15min":1.137},"max":{"1min":2.254,"5min":2.254,"15min":2.254},"last":1.361}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.988}]},{"osd":8,"last update":"Sat Apr 2 03:16:54 2022","interfaces":[{"interface":"back","average":{"1min":1.7150000000000001,"5min":1.7150000000000001,"15min":1.7150000000000001},"min":{"1min":1.03,"5min":1.03,"15min":1.03},"max":{"1min":2.5390000000000001,"5min":2.5390000000000001,"15min":2.5390000000000001},"last":1.6619999999999999},{"interface":"front","average":{"1min":1.71,"5min":1.71,"15min":1.71},"min":{"1min":1.395,"5min":1.395,"15min":1.395},"max":{"1min":2.0609999999999999,"5min":2.0609999999999999,"15min":2.0609999999999999},"last":1.651}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8560000000000001}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.903}]}]},{"osd":4,"up_from":20,"seq":85899345942,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5368,"kb_used_data":368,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":93739784,"statfs":{"total":95995035648,"available":95989538816,"internally_reserved":0,"allocated":376832,"data_stored":166907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 2 03:16:46 2022","interfaces":[{"interface":"back","average":{"1min":1.2989999999999999,"5min":1.2989999999999999,"15min":1.2989999999999999},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":2.056,"5min":2.056,"15min":2.056},"last":1.3899999999999999},{"interface":"front","average":{"1min":1.202,"5min":1.202,"15min":1.202},"min":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"max":{"1min":1.9079999999999999,"5min":1.9079999999999999,"15min":1.9079999999999999},"last":1.514}]},{"osd":1,"last update":"Sat Apr 2 03:16:46 2022","interfaces":[{"interface":"back","average":{"1min":1.395,"5min":1.395,"15min":1.395},"min":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"max":{"1min":1.9359999999999999,"5min":1.9359999999999999,"15min":1.9359999999999999},"last":2.2389999999999999},{"interface":"front","average":{"1min":1.4259999999999999,"5min":1.4259999999999999,"15min":1.4259999999999999},"min":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"max":{"1min":2.1099999999999999,"5min":2.1099999999999999,"15min":2.1099999999999999},"last":1.8009999999999999}]},{"osd":2,"last update":"Sat Apr 2 03:16:46 2022","interfaces":[{"interface":"back","average":{"1min":1.2749999999999999,"5min":1.2749999999999999,"15min":1.2749999999999999},"min":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"max":{"1min":1.9019999999999999,"5min":1.9019999999999999,"15min":1.9019999999999999},"last":1.552},{"interface":"front","average":{"1min":1.48,"5min":1.48,"15min":1.48},"min":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"max":{"1min":2.2349999999999999,"5min":2.2349999999999999,"15min":2.2349999999999999},"last":1.452}]},{"osd":3,"last update":"Sat Apr 2 03:16:53 2022","interfaces":[{"interface":"back","average":{"1min":1.1080000000000001,"5min":1.1080000000000001,"15min":1.1080000000000001},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.7430000000000001,"5min":1.7430000000000001,"15min":1.7430000000000001},"last":1.6910000000000001},{"interface":"front","average":{"1min":1.2669999999999999,"5min":1.2669999999999999,"15min":1.2669999999999999},"min":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"max":{"1min":1.9139999999999999,"5min":1.9139999999999999,"15min":1.9139999999999999},"last":1.319}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8640000000000001}]},{"osd":6,"last update":"Sat Apr 2 03:16:53 2022","interfaces":[{"interface":"back","average":{"1min":1.5920000000000001,"5min":1.5920000000000001,"15min":1.5920000000000001},"min":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":2.3170000000000002,"5min":2.3170000000000002,"15min":2.3170000000000002},"last":2.706},{"interface":"front","average":{"1min":1.5609999999999999,"5min":1.5609999999999999,"15min":1.5609999999999999},"min":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":2.7989999999999999,"5min":2.7989999999999999,"15min":2.7989999999999999},"last":1.986}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.8140000000000001}]},{"osd":8,"last update":"Sat Apr 2 03:16:53 2022","interfaces":[{"interface":"back","average":{"1min":1.621,"5min":1.621,"15min":1.621},"min":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"max":{"1min":2.375,"5min":2.375,"15min":2.375},"last":1.669},{"interface":"front","average":{"1min":1.831,"5min":1.831,"15min":1.831},"min":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"max":{"1min":3.1389999999999998,"5min":3.1389999999999998,"15min":3.1389999999999998},"last":2.964}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.8980000000000001}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.339}]}]},{"osd":3,"up_from":22,"seq":94489280530,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6400,"kb_used_data":888,"kb_used_omap":0,"kb_used_meta":5504,"kb_avail":93738752,"statfs":{"total":95995035648,"available":95988482048,"internally_reserved":0,"allocated":909312,"data_stored":695835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5636096},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 2 03:16:48 2022","interfaces":[{"interface":"back","average":{"1min":1.3200000000000001,"5min":1.3200000000000001,"15min":1.3200000000000001},"min":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":2.1349999999999998,"5min":2.1349999999999998,"15min":2.1349999999999998},"last":1.5589999999999999},{"interface":"front","average":{"1min":1.3540000000000001,"5min":1.3540000000000001,"15min":1.3540000000000001},"min":{"1min":0.747,"5min":0.747,"15min":0.747},"max":{"1min":2.2629999999999999,"5min":2.2629999999999999,"15min":2.2629999999999999},"last":1.248}]},{"osd":1,"last update":"Sat Apr 2 03:16:48 2022","interfaces":[{"interface":"back","average":{"1min":1.335,"5min":1.335,"15min":1.335},"min":{"1min":1,"5min":1,"15min":1},"max":{"1min":2.258,"5min":2.258,"15min":2.258},"last":1.417},{"interface":"front","average":{"1min":1.3280000000000001,"5min":1.3280000000000001,"15min":1.3280000000000001},"min":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"max":{"1min":3.1869999999999998,"5min":3.1869999999999998,"15min":3.1869999999999998},"last":1.353}]},{"osd":2,"last update":"Sat Apr 2 03:16:48 2022","interfaces":[{"interface":"back","average":{"1min":1.4530000000000001,"5min":1.4530000000000001,"15min":1.4530000000000001},"min":{"1min":1.143,"5min":1.143,"15min":1.143},"max":{"1min":2.3860000000000001,"5min":2.3860000000000001,"15min":2.3860000000000001},"last":1.677},{"interface":"front","average":{"1min":1.621,"5min":1.621,"15min":1.621},"min":{"1min":1.141,"5min":1.141,"15min":1.141},"max":{"1min":5.0439999999999996,"5min":5.0439999999999996,"15min":5.0439999999999996},"last":1.853}]},{"osd":4,"last update":"Sat Apr 2 03:16:48 2022","interfaces":[{"interface":"back","average":{"1min":1.2849999999999999,"5min":1.2849999999999999,"15min":1.2849999999999999},"min":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":3.0800000000000001,"5min":3.0800000000000001,"15min":3.0800000000000001},"last":1.774},{"interface":"front","average":{"1min":1.254,"5min":1.254,"15min":1.254},"min":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"max":{"1min":1.8500000000000001,"5min":1.8500000000000001,"15min":1.8500000000000001},"last":2.0539999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5900000000000001}]},{"osd":6,"last update":"Sat Apr 2 03:16:51 2022","interfaces":[{"interface":"back","average":{"1min":1.446,"5min":1.446,"15min":1.446},"min":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"max":{"1min":3.1389999999999998,"5min":3.1389999999999998,"15min":3.1389999999999998},"last":1.0589999999999999},{"interface":"front","average":{"1min":1.581,"5min":1.581,"15min":1.581},"min":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"max":{"1min":2.4020000000000001,"5min":2.4020000000000001,"15min":2.4020000000000001},"last":1.7430000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8859999999999999}]},{"osd":8,"last update":"Sat Apr 2 03:16:48 2022","interfaces":[{"interface":"back","average":{"1min":1.849,"5min":1.849,"15min":1.849},"min":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"max":{"1min":5.5519999999999996,"5min":5.5519999999999996,"15min":5.5519999999999996},"last":1.724},{"interface":"front","average":{"1min":1.681,"5min":1.681,"15min":1.681},"min":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"max":{"1min":3.3919999999999999,"5min":3.3919999999999999,"15min":3.3919999999999999},"last":1.9199999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.387}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.536}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2022-04-02T03:17:04.670 INFO:tasks.rook.ceph_manager.ceph:clean! 2022-04-02T03:17:04.670 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2022-04-02T03:17:04.671 INFO:tasks.rook.ceph_manager.ceph:wait_until_healthy 2022-04-02T03:17:04.672 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph health --format=json 2022-04-02T03:17:05.225 INFO:teuthology.orchestra.run.smithi029.stdout: 2022-04-02T03:17:05.225 INFO:teuthology.orchestra.run.smithi029.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2022-04-02T03:17:05.247 INFO:tasks.rook.ceph_manager.ceph:wait_until_healthy done 2022-04-02T03:17:05.248 INFO:tasks.rook:Rook complete, yielding 2022-04-02T03:17:05.248 INFO:teuthology.run_tasks:Running task rook.shell... 2022-04-02T03:17:05.264 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph -s 2022-04-02T03:17:05.849 INFO:teuthology.orchestra.run.smithi029.stdout: cluster: 2022-04-02T03:17:05.850 INFO:teuthology.orchestra.run.smithi029.stdout: id: 17047c32-2dfc-4a19-8df4-d58d43ddcdff 2022-04-02T03:17:05.850 INFO:teuthology.orchestra.run.smithi029.stdout: health: HEALTH_OK 2022-04-02T03:17:05.850 INFO:teuthology.orchestra.run.smithi029.stdout: 2022-04-02T03:17:05.850 INFO:teuthology.orchestra.run.smithi029.stdout: services: 2022-04-02T03:17:05.851 INFO:teuthology.orchestra.run.smithi029.stdout: mon: 3 daemons, quorum a,b,c (age 3m) 2022-04-02T03:17:05.851 INFO:teuthology.orchestra.run.smithi029.stdout: mgr: a(active, since 3m) 2022-04-02T03:17:05.851 INFO:teuthology.orchestra.run.smithi029.stdout: osd: 12 osds: 12 up (since 21s), 12 in (since 113s) 2022-04-02T03:17:05.851 INFO:teuthology.orchestra.run.smithi029.stdout: 2022-04-02T03:17:05.851 INFO:teuthology.orchestra.run.smithi029.stdout: data: 2022-04-02T03:17:05.852 INFO:teuthology.orchestra.run.smithi029.stdout: pools: 1 pools, 1 pgs 2022-04-02T03:17:05.852 INFO:teuthology.orchestra.run.smithi029.stdout: objects: 2 objects, 577 KiB 2022-04-02T03:17:05.852 INFO:teuthology.orchestra.run.smithi029.stdout: usage: 66 MiB used, 1.0 TiB / 1.0 TiB avail 2022-04-02T03:17:05.852 INFO:teuthology.orchestra.run.smithi029.stdout: pgs: 1 active+clean 2022-04-02T03:17:05.853 INFO:teuthology.orchestra.run.smithi029.stdout: 2022-04-02T03:17:05.870 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph orch status 2022-04-02T03:17:06.576 INFO:teuthology.orchestra.run.smithi029.stdout:Backend: rook 2022-04-02T03:17:06.577 INFO:teuthology.orchestra.run.smithi029.stdout:Available: Yes 2022-04-02T03:17:06.603 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph orch ps 2022-04-02T03:17:07.195 INFO:teuthology.orchestra.run.smithi029.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID 2022-04-02T03:17:07.196 INFO:teuthology.orchestra.run.smithi029.stdout:crashcollector.crash smithi029 running (113s) 0s ago 113s - - a1d2ec729cd2 2022-04-02T03:17:07.196 INFO:teuthology.orchestra.run.smithi029.stdout:crashcollector.crash smithi104 running (3m) 0s ago 3m - - a1d2ec729cd2 2022-04-02T03:17:07.196 INFO:teuthology.orchestra.run.smithi029.stdout:crashcollector.crash smithi142 running (3m) 0s ago 3m - - a1d2ec729cd2 2022-04-02T03:17:07.196 INFO:teuthology.orchestra.run.smithi029.stdout:mgr.a smithi029 running (3m) 0s ago 3m - - a1d2ec729cd2 2022-04-02T03:17:07.197 INFO:teuthology.orchestra.run.smithi029.stdout:mon.a smithi104 running (7m) 0s ago 7m - - a1d2ec729cd2 2022-04-02T03:17:07.197 INFO:teuthology.orchestra.run.smithi029.stdout:mon.b smithi142 running (5m) 0s ago 5m - - a1d2ec729cd2 2022-04-02T03:17:07.197 INFO:teuthology.orchestra.run.smithi029.stdout:mon.c smithi029 running (4m) 0s ago 4m - - a1d2ec729cd2 2022-04-02T03:17:07.197 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0 smithi104 running (2m) 0s ago 2m - - a1d2ec729cd2 2022-04-02T03:17:07.198 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1 smithi104 running (119s) 0s ago 119s - - a1d2ec729cd2 2022-04-02T03:17:07.198 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2 smithi104 running (115s) 0s ago 115s - - a1d2ec729cd2 2022-04-02T03:17:07.198 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3 smithi142 running (115s) 0s ago 115s - - a1d2ec729cd2 2022-04-02T03:17:07.198 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4 smithi142 running (115s) 0s ago 115s - - a1d2ec729cd2 2022-04-02T03:17:07.199 INFO:teuthology.orchestra.run.smithi029.stdout:osd.5 smithi029 running (114s) 0s ago 114s - - a1d2ec729cd2 2022-04-02T03:17:07.199 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6 smithi142 running (110s) 0s ago 110s - - a1d2ec729cd2 2022-04-02T03:17:07.199 INFO:teuthology.orchestra.run.smithi029.stdout:osd.7 smithi029 running (108s) 0s ago 109s - - a1d2ec729cd2 2022-04-02T03:17:07.199 INFO:teuthology.orchestra.run.smithi029.stdout:osd.8 smithi142 running (107s) 0s ago 107s - - a1d2ec729cd2 2022-04-02T03:17:07.199 INFO:teuthology.orchestra.run.smithi029.stdout:osd.9 smithi029 running (104s) 0s ago 105s - - a1d2ec729cd2 2022-04-02T03:17:07.200 INFO:teuthology.orchestra.run.smithi029.stdout:osd.10 smithi029 running (102s) 0s ago 102s - - a1d2ec729cd2 2022-04-02T03:17:07.200 INFO:teuthology.orchestra.run.smithi029.stdout:osd.11 smithi104 running (43s) 0s ago 43s - - a1d2ec729cd2 2022-04-02T03:17:07.222 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph orch ls 2022-04-02T03:17:07.963 INFO:teuthology.orchestra.run.smithi029.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2022-04-02T03:17:07.964 INFO:teuthology.orchestra.run.smithi029.stdout:crash 3/3 0s ago 3m * 2022-04-02T03:17:07.964 INFO:teuthology.orchestra.run.smithi029.stdout:mgr 1/1 0s ago 3m count:1 2022-04-02T03:17:07.964 INFO:teuthology.orchestra.run.smithi029.stdout:mon 3/3 0s ago 7m count:3 2022-04-02T03:17:07.964 INFO:teuthology.orchestra.run.smithi029.stdout:osd 24 0s ago 2m 2022-04-02T03:17:07.965 INFO:teuthology.orchestra.run.smithi029.stdout:osd.all-available-devices 0 0s ago - * 2022-04-02T03:17:07.987 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph orch device ls 2022-04-02T03:17:08.536 INFO:teuthology.orchestra.run.smithi029.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2022-04-02T03:17:08.536 INFO:teuthology.orchestra.run.smithi029.stdout:smithi029 /dev/vg_nvme/lv_1 unknown None 107G 0s ago 2022-04-02T03:17:08.537 INFO:teuthology.orchestra.run.smithi029.stdout:smithi029 /dev/vg_nvme/lv_2 unknown None 107G 0s ago 2022-04-02T03:17:08.537 INFO:teuthology.orchestra.run.smithi029.stdout:smithi029 /dev/vg_nvme/lv_3 unknown None 107G 0s ago 2022-04-02T03:17:08.537 INFO:teuthology.orchestra.run.smithi029.stdout:smithi029 /dev/vg_nvme/lv_4 unknown None 107G 0s ago 2022-04-02T03:17:08.537 INFO:teuthology.orchestra.run.smithi029.stdout:smithi104 /dev/vg_nvme/lv_1 unknown None 107G 0s ago 2022-04-02T03:17:08.538 INFO:teuthology.orchestra.run.smithi029.stdout:smithi104 /dev/vg_nvme/lv_2 unknown None 107G 0s ago 2022-04-02T03:17:08.538 INFO:teuthology.orchestra.run.smithi029.stdout:smithi104 /dev/vg_nvme/lv_3 unknown None 107G 0s ago 2022-04-02T03:17:08.538 INFO:teuthology.orchestra.run.smithi029.stdout:smithi104 /dev/vg_nvme/lv_4 unknown None 107G 0s ago 2022-04-02T03:17:08.538 INFO:teuthology.orchestra.run.smithi029.stdout:smithi142 /dev/vg_nvme/lv_1 unknown None 107G 0s ago 2022-04-02T03:17:08.539 INFO:teuthology.orchestra.run.smithi029.stdout:smithi142 /dev/vg_nvme/lv_2 unknown None 107G 0s ago 2022-04-02T03:17:08.539 INFO:teuthology.orchestra.run.smithi029.stdout:smithi142 /dev/vg_nvme/lv_3 unknown None 107G 0s ago 2022-04-02T03:17:08.539 INFO:teuthology.orchestra.run.smithi029.stdout:smithi142 /dev/vg_nvme/lv_4 unknown None 107G 0s ago 2022-04-02T03:17:08.560 INFO:teuthology.run_tasks:Running task exec... 2022-04-02T03:17:08.572 INFO:teuthology.task.exec:Executing custom commands... 2022-04-02T03:17:08.572 INFO:teuthology.task.exec:Running commands on role host.a host ubuntu@smithi029.front.sepia.ceph.com 2022-04-02T03:17:08.573 DEBUG:teuthology.orchestra.run.smithi029:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'set -ex 2022-04-02T03:17:08.573 DEBUG:teuthology.orchestra.run.smithi029:> toolbox() { 2022-04-02T03:17:08.573 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- "$@" 2022-04-02T03:17:08.574 DEBUG:teuthology.orchestra.run.smithi029:> } 2022-04-02T03:17:08.574 DEBUG:teuthology.orchestra.run.smithi029:> orig_num_osd=`toolbox ceph osd stat | cut -f3 -d " "` 2022-04-02T03:17:08.574 DEBUG:teuthology.orchestra.run.smithi029:> toolbox ceph orch osd rm 0 --force 2022-04-02T03:17:08.575 DEBUG:teuthology.orchestra.run.smithi029:> removed_pv="" 2022-04-02T03:17:08.575 DEBUG:teuthology.orchestra.run.smithi029:> while [ "$removed_pv" = "" ] 2022-04-02T03:17:08.575 DEBUG:teuthology.orchestra.run.smithi029:> do 2022-04-02T03:17:08.575 DEBUG:teuthology.orchestra.run.smithi029:> removed_pv=`kubectl get pv | grep Released | cut -f1 -d " "` 2022-04-02T03:17:08.576 DEBUG:teuthology.orchestra.run.smithi029:> sleep 3s 2022-04-02T03:17:08.576 DEBUG:teuthology.orchestra.run.smithi029:> done 2022-04-02T03:17:08.576 DEBUG:teuthology.orchestra.run.smithi029:> target_path=`kubectl get pv $removed_pv -o jsonpath='"'"'{.spec.local.path}'"'"'` 2022-04-02T03:17:08.576 DEBUG:teuthology.orchestra.run.smithi029:> host=`echo $removed_pv | cut -f1 -d "-"` 2022-04-02T03:17:08.577 DEBUG:teuthology.orchestra.run.smithi029:> toolbox ceph orch device zap $host $target_path --force 2022-04-02T03:17:08.577 DEBUG:teuthology.orchestra.run.smithi029:> zap_completion="0" 2022-04-02T03:17:08.577 DEBUG:teuthology.orchestra.run.smithi029:> while [ "$zap_completion" = "0" ] 2022-04-02T03:17:08.578 DEBUG:teuthology.orchestra.run.smithi029:> do 2022-04-02T03:17:08.578 DEBUG:teuthology.orchestra.run.smithi029:> zap_completion=`kubectl get job -n rook-ceph rook-ceph-device-zap -o jsonpath='"'"'{.status.succeeded.path}'"'"'` 2022-04-02T03:17:08.578 DEBUG:teuthology.orchestra.run.smithi029:> sleep 3s 2022-04-02T03:17:08.578 DEBUG:teuthology.orchestra.run.smithi029:> done 2022-04-02T03:17:08.579 DEBUG:teuthology.orchestra.run.smithi029:> kubectl patch pv $removed_pv -p '"'"'{"spec":{"claimRef": null}}'"'"' 2022-04-02T03:17:08.579 DEBUG:teuthology.orchestra.run.smithi029:> toolbox ceph orch apply osd --all-available-devices 2022-04-02T03:17:08.579 DEBUG:teuthology.orchestra.run.smithi029:> kubectl delete job rook-ceph-device-zap -n rook-ceph 2022-04-02T03:17:08.579 DEBUG:teuthology.orchestra.run.smithi029:> num_osd="0" 2022-04-02T03:17:08.580 DEBUG:teuthology.orchestra.run.smithi029:> while [ "$num_osd" != "$orig_num_osd" ] 2022-04-02T03:17:08.580 DEBUG:teuthology.orchestra.run.smithi029:> do 2022-04-02T03:17:08.580 DEBUG:teuthology.orchestra.run.smithi029:> echo "waiting for osd to come back up" 2022-04-02T03:17:08.581 DEBUG:teuthology.orchestra.run.smithi029:> num_osd=`toolbox ceph osd stat | cut -f3 -d " "` 2022-04-02T03:17:08.581 DEBUG:teuthology.orchestra.run.smithi029:> sleep 30s 2022-04-02T03:17:08.581 DEBUG:teuthology.orchestra.run.smithi029:> done 2022-04-02T03:17:08.581 DEBUG:teuthology.orchestra.run.smithi029:> ' 2022-04-02T03:17:08.598 INFO:teuthology.orchestra.run.smithi029.stderr:++ toolbox ceph osd stat 2022-04-02T03:17:08.599 INFO:teuthology.orchestra.run.smithi029.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-04-02T03:17:08.599 INFO:teuthology.orchestra.run.smithi029.stderr:++ cut -f3 -d ' ' 2022-04-02T03:17:11.420 INFO:teuthology.orchestra.run.smithi029.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-04-02T03:17:11.921 INFO:teuthology.orchestra.run.smithi029.stderr:+ orig_num_osd=12 2022-04-02T03:17:11.921 INFO:teuthology.orchestra.run.smithi029.stderr:+ toolbox ceph orch osd rm 0 --force 2022-04-02T03:17:11.921 INFO:teuthology.orchestra.run.smithi029.stderr:+ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph orch osd rm 0 --force 2022-04-02T03:17:12.000 INFO:teuthology.orchestra.run.smithi029.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-04-02T03:17:12.624 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:12.617516 I | ceph-cluster-controller: CR has changed for "rook-ceph". diff= v1.ClusterSpec{ 2022-04-02T03:17:12.625 INFO:tasks.rook.operator.smithi029.stdout: CephVersion: {Image: "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", AllowUnsupported: true}, 2022-04-02T03:17:12.625 INFO:tasks.rook.operator.smithi029.stdout: Storage: v1.StorageScopeSpec{ 2022-04-02T03:17:12.625 INFO:tasks.rook.operator.smithi029.stdout: ... // 3 identical fields 2022-04-02T03:17:12.626 INFO:tasks.rook.operator.smithi029.stdout: Config: nil, 2022-04-02T03:17:12.626 INFO:tasks.rook.operator.smithi029.stdout: Selection: {}, 2022-04-02T03:17:12.626 INFO:tasks.rook.operator.smithi029.stdout: StorageClassDeviceSets: []v1.StorageClassDeviceSet{ 2022-04-02T03:17:12.626 INFO:tasks.rook.operator.smithi029.stdout: ... // 2 identical elements 2022-04-02T03:17:12.627 INFO:tasks.rook.operator.smithi029.stdout: {Name: "smithi029-lv-3", Count: 1, VolumeClaimTemplates: {{ObjectMeta: {Name: "data"}, Spec: {AccessModes: {"ReadWriteOnce"}, Resources: {Requests: {s"storage": {i: {...}, s: "1", Format: "DecimalSI"}}}, VolumeName: "smithi029-lv-3", StorageClassName: &"scratch", ...}}}}, 2022-04-02T03:17:12.627 INFO:tasks.rook.operator.smithi029.stdout: {Name: "smithi029-lv-4", Count: 1, VolumeClaimTemplates: {{ObjectMeta: {Name: "data"}, Spec: {AccessModes: {"ReadWriteOnce"}, Resources: {Requests: {s"storage": {i: {...}, s: "1", Format: "DecimalSI"}}}, VolumeName: "smithi029-lv-4", StorageClassName: &"scratch", ...}}}}, 2022-04-02T03:17:12.627 INFO:tasks.rook.operator.smithi029.stdout:- { 2022-04-02T03:17:12.627 INFO:tasks.rook.operator.smithi029.stdout:- Name: "smithi104-lv-1", 2022-04-02T03:17:12.627 INFO:tasks.rook.operator.smithi029.stdout:- Count: 1, 2022-04-02T03:17:12.628 INFO:tasks.rook.operator.smithi029.stdout:- VolumeClaimTemplates: []v1.PersistentVolumeClaim{{ObjectMeta: v1.ObjectMeta{...}, Spec: v1.PersistentVolumeClaimSpec{...}}}, 2022-04-02T03:17:12.628 INFO:tasks.rook.operator.smithi029.stdout:- }, 2022-04-02T03:17:12.628 INFO:tasks.rook.operator.smithi029.stdout: {Name: "smithi104-lv-2", Count: 1, VolumeClaimTemplates: {{ObjectMeta: {Name: "data"}, Spec: {AccessModes: {"ReadWriteOnce"}, Resources: {Requests: {s"storage": {i: {...}, s: "1", Format: "DecimalSI"}}}, VolumeName: "smithi104-lv-2", StorageClassName: &"scratch", ...}}}}, 2022-04-02T03:17:12.628 INFO:tasks.rook.operator.smithi029.stdout: {Name: "smithi104-lv-3", Count: 1, VolumeClaimTemplates: {{ObjectMeta: {Name: "data"}, Spec: {AccessModes: {"ReadWriteOnce"}, Resources: {Requests: {s"storage": {i: {...}, s: "1", Format: "DecimalSI"}}}, VolumeName: "smithi104-lv-3", StorageClassName: &"scratch", ...}}}}, 2022-04-02T03:17:12.629 INFO:tasks.rook.operator.smithi029.stdout: ... // 5 identical elements 2022-04-02T03:17:12.629 INFO:tasks.rook.operator.smithi029.stdout: }, 2022-04-02T03:17:12.629 INFO:tasks.rook.operator.smithi029.stdout: }, 2022-04-02T03:17:12.629 INFO:tasks.rook.operator.smithi029.stdout: Annotations: nil, 2022-04-02T03:17:12.629 INFO:tasks.rook.operator.smithi029.stdout: Labels: nil, 2022-04-02T03:17:12.630 INFO:tasks.rook.operator.smithi029.stdout: ... // 20 identical fields 2022-04-02T03:17:12.630 INFO:tasks.rook.operator.smithi029.stdout: } 2022-04-02T03:17:12.630 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:12.617714 I | operator: reloading operator's CRDs manager, cancelling all orchestrations! 2022-04-02T03:17:12.630 INFO:tasks.rook.operator.smithi029.stdout:I0402 03:17:12.618010 1 manager.go:148] objectbucket.io/provisioner-manager "msg"="stopping provisioner" "name"="rook-ceph.ceph.rook.io/bucket" "reason"={} 2022-04-02T03:17:12.631 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:12.618070 I | ceph-cluster-controller: stopping monitoring of ceph status 2022-04-02T03:17:12.631 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:12.618093 I | operator: successfully started the controller-runtime manager 2022-04-02T03:17:12.631 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:12.618107 I | op-mon: stopping monitoring of mons in namespace "rook-ceph" 2022-04-02T03:17:12.631 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:12.618115 I | op-osd: stopping monitoring of OSDs in namespace "rook-ceph" 2022-04-02T03:17:12.679 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:12.674180 I | op-k8sutil: ROOK_CURRENT_NAMESPACE_ONLY="false" (env var) 2022-04-02T03:17:12.679 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:12.674196 I | operator: watching all namespaces for Ceph CRs 2022-04-02T03:17:12.679 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:12.674222 I | operator: setting up schemes 2022-04-02T03:17:12.683 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:12.675926 I | operator: setting up the controller-runtime manager 2022-04-02T03:17:12.884 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:12.879510 I | operator: looking for admission webhook secret "rook-ceph-admission-controller" 2022-04-02T03:17:12.944 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:12.938485 I | operator: admission webhook secret "rook-ceph-admission-controller" not found. proceeding without the admission controller 2022-04-02T03:17:12.946 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:12.938527 I | ceph-cluster-controller: successfully started 2022-04-02T03:17:12.946 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:12.938587 I | ceph-cluster-controller: enabling hotplug orchestration 2022-04-02T03:17:12.947 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:12.938609 I | ceph-crashcollector-controller: successfully started 2022-04-02T03:17:12.947 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:12.938650 I | ceph-block-pool-controller: successfully started 2022-04-02T03:17:12.947 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:12.938682 I | ceph-object-store-user-controller: successfully started 2022-04-02T03:17:12.947 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:12.938715 I | ceph-object-realm-controller: successfully started 2022-04-02T03:17:12.947 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:12.938733 I | ceph-object-zonegroup-controller: successfully started 2022-04-02T03:17:12.948 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:12.938751 I | ceph-object-zone-controller: successfully started 2022-04-02T03:17:12.948 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:12.938984 I | ceph-object-controller: successfully started 2022-04-02T03:17:12.949 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:12.939051 I | ceph-file-controller: successfully started 2022-04-02T03:17:12.950 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:12.939113 I | ceph-nfs-controller: successfully started 2022-04-02T03:17:12.950 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:12.939165 I | ceph-rbd-mirror-controller: successfully started 2022-04-02T03:17:12.950 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:12.939217 I | ceph-client-controller: successfully started 2022-04-02T03:17:12.950 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:12.939265 I | ceph-filesystem-mirror-controller: successfully started 2022-04-02T03:17:12.951 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:12.939321 I | operator: rook-ceph-operator-config-controller successfully started 2022-04-02T03:17:12.951 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:12.939359 I | ceph-csi: rook-ceph-operator-csi-controller successfully started 2022-04-02T03:17:12.951 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:12.939393 I | op-bucket-prov: rook-ceph-operator-bucket-controller successfully started 2022-04-02T03:17:12.951 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:12.939429 I | ceph-bucket-topic: successfully started 2022-04-02T03:17:12.952 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:12.939457 I | ceph-bucket-notification: successfully started 2022-04-02T03:17:12.952 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:12.939484 I | ceph-bucket-notification: successfully started 2022-04-02T03:17:12.952 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:12.939509 I | ceph-fs-subvolumegroup-controller: successfully started 2022-04-02T03:17:12.952 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:12.946396 I | operator: starting the controller-runtime manager 2022-04-02T03:17:13.061 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:13.054652 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-04-02T03:17:13.061 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:13.055447 I | op-k8sutil: ROOK_CEPH_COMMANDS_TIMEOUT_SECONDS="15" (configmap) 2022-04-02T03:17:13.061 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:13.055519 I | op-k8sutil: ROOK_LOG_LEVEL="INFO" (configmap) 2022-04-02T03:17:13.062 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:13.055541 I | op-k8sutil: ROOK_ENABLE_DISCOVERY_DAEMON="false" (configmap) 2022-04-02T03:17:13.117 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:13.112322 I | operator: rook-ceph-operator-config-controller done reconciling 2022-04-02T03:17:13.120 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:13.113294 I | op-mon: parsing mon endpoints: a=10.102.133.199:6789,b=10.107.147.59:6789,c=10.96.208.36:6789 2022-04-02T03:17:13.121 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:13.113336 I | op-k8sutil: ROOK_OBC_WATCH_OPERATOR_NAMESPACE="true" (configmap) 2022-04-02T03:17:13.121 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:13.113346 I | op-bucket-prov: ceph bucket provisioner launched watching for provisioner "rook-ceph.ceph.rook.io/bucket" 2022-04-02T03:17:13.121 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:13.114286 I | op-bucket-prov: successfully reconciled bucket provisioner 2022-04-02T03:17:13.121 INFO:tasks.rook.operator.smithi029.stdout:I0402 03:17:13.114337 1 manager.go:135] objectbucket.io/provisioner-manager "msg"="starting provisioner" "name"="rook-ceph.ceph.rook.io/bucket" 2022-04-02T03:17:13.125 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:13.121757 I | op-mon: parsing mon endpoints: a=10.102.133.199:6789,b=10.107.147.59:6789,c=10.96.208.36:6789 2022-04-02T03:17:13.143 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:13.138469 I | ceph-csi: successfully created csi config map "rook-ceph-csi-config" 2022-04-02T03:17:13.147 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:13.140794 I | ceph-cluster-controller: enabling ceph mon monitoring goroutine for cluster "rook-ceph" 2022-04-02T03:17:13.147 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:13.140826 I | ceph-cluster-controller: enabling ceph osd monitoring goroutine for cluster "rook-ceph" 2022-04-02T03:17:13.148 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:13.140837 I | ceph-cluster-controller: enabling ceph status monitoring goroutine for cluster "rook-ceph" 2022-04-02T03:17:13.176 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:13.172223 I | ceph-spec: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a... 2022-04-02T03:17:13.210 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:13.205483 I | op-k8sutil: ROOK_CSI_ENABLE_RBD="true" (configmap) 2022-04-02T03:17:13.212 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:13.205517 I | op-k8sutil: ROOK_CSI_ENABLE_CEPHFS="true" (configmap) 2022-04-02T03:17:13.212 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:13.205527 I | op-k8sutil: ROOK_CSI_ALLOW_UNSUPPORTED_VERSION="false" (configmap) 2022-04-02T03:17:13.212 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:13.205536 I | op-k8sutil: ROOK_CSI_ENABLE_GRPC_METRICS="false" (configmap) 2022-04-02T03:17:13.213 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:13.205546 I | op-k8sutil: CSI_ENABLE_HOST_NETWORK="true" (default) 2022-04-02T03:17:13.213 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:13.205555 I | op-k8sutil: ROOK_CSI_CEPH_IMAGE="quay.io/cephcsi/cephcsi:v3.5.1" (default) 2022-04-02T03:17:13.213 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:13.205565 I | op-k8sutil: ROOK_CSI_REGISTRAR_IMAGE="k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.5.0" (default) 2022-04-02T03:17:13.214 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:13.205575 I | op-k8sutil: ROOK_CSI_PROVISIONER_IMAGE="k8s.gcr.io/sig-storage/csi-provisioner:v3.1.0" (default) 2022-04-02T03:17:13.214 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:13.205584 I | op-k8sutil: ROOK_CSI_ATTACHER_IMAGE="k8s.gcr.io/sig-storage/csi-attacher:v3.4.0" (default) 2022-04-02T03:17:13.215 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:13.205590 I | op-k8sutil: ROOK_CSI_SNAPSHOTTER_IMAGE="k8s.gcr.io/sig-storage/csi-snapshotter:v5.0.1" (default) 2022-04-02T03:17:13.215 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:13.205596 I | op-k8sutil: ROOK_CSI_KUBELET_DIR_PATH="/var/lib/kubelet" (default) 2022-04-02T03:17:13.215 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:13.205601 I | op-k8sutil: CSI_VOLUME_REPLICATION_IMAGE="quay.io/csiaddons/volumereplication-operator:v0.3.0" (default) 2022-04-02T03:17:13.215 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:13.205607 I | op-k8sutil: ROOK_CSIADDONS_IMAGE="quay.io/csiaddons/k8s-sidecar:v0.2.1" (default) 2022-04-02T03:17:13.216 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:13.205612 I | op-k8sutil: ROOK_CSI_CEPHFS_POD_LABELS="" (default) 2022-04-02T03:17:13.216 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:13.205617 I | op-k8sutil: ROOK_CSI_RBD_POD_LABELS="" (default) 2022-04-02T03:17:13.216 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:13.205623 I | ceph-csi: detecting the ceph csi image version for image "quay.io/cephcsi/cephcsi:v3.5.1" 2022-04-02T03:17:13.216 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:13.205689 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-04-02T03:17:13.217 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:13.205700 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default) 2022-04-02T03:17:19.023 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:19.018588 I | ceph-spec: object "rook-ceph-osd-0" matched on delete, reconciling 2022-04-02T03:17:19.226 INFO:teuthology.orchestra.run.smithi029.stdout:Success 2022-04-02T03:17:19.255 INFO:teuthology.orchestra.run.smithi029.stderr:+ removed_pv= 2022-04-02T03:17:19.255 INFO:teuthology.orchestra.run.smithi029.stderr:+ '[' '' = '' ']' 2022-04-02T03:17:19.256 INFO:teuthology.orchestra.run.smithi029.stderr:++ kubectl get pv 2022-04-02T03:17:19.257 INFO:teuthology.orchestra.run.smithi029.stderr:++ grep Released 2022-04-02T03:17:19.257 INFO:teuthology.orchestra.run.smithi029.stderr:++ cut -f1 -d ' ' 2022-04-02T03:17:19.310 INFO:teuthology.orchestra.run.smithi029.stderr:+ removed_pv=smithi104-lv-1 2022-04-02T03:17:19.310 INFO:teuthology.orchestra.run.smithi029.stderr:+ sleep 3s 2022-04-02T03:17:22.311 INFO:teuthology.orchestra.run.smithi029.stderr:+ '[' smithi104-lv-1 = '' ']' 2022-04-02T03:17:22.311 INFO:teuthology.orchestra.run.smithi029.stderr:++ kubectl get pv smithi104-lv-1 -o 'jsonpath={.spec.local.path}' 2022-04-02T03:17:22.380 INFO:teuthology.orchestra.run.smithi029.stderr:+ target_path=/dev/vg_nvme/lv_1 2022-04-02T03:17:22.380 INFO:teuthology.orchestra.run.smithi029.stderr:++ echo smithi104-lv-1 2022-04-02T03:17:22.381 INFO:teuthology.orchestra.run.smithi029.stderr:++ cut -f1 -d - 2022-04-02T03:17:22.382 INFO:teuthology.orchestra.run.smithi029.stderr:+ host=smithi104 2022-04-02T03:17:22.382 INFO:teuthology.orchestra.run.smithi029.stderr:+ toolbox ceph orch device zap smithi104 /dev/vg_nvme/lv_1 --force 2022-04-02T03:17:22.383 INFO:teuthology.orchestra.run.smithi029.stderr:+ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph orch device zap smithi104 /dev/vg_nvme/lv_1 --force 2022-04-02T03:17:22.459 INFO:teuthology.orchestra.run.smithi029.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-04-02T03:17:23.019 INFO:teuthology.orchestra.run.smithi029.stdout:/dev/vg_nvme/lv_1 on smithi104 zapped 2022-04-02T03:17:23.041 INFO:teuthology.orchestra.run.smithi029.stderr:+ zap_completion=0 2022-04-02T03:17:23.042 INFO:teuthology.orchestra.run.smithi029.stderr:+ '[' 0 = 0 ']' 2022-04-02T03:17:23.042 INFO:teuthology.orchestra.run.smithi029.stderr:++ kubectl get job -n rook-ceph rook-ceph-device-zap -o 'jsonpath={.status.succeeded.path}' 2022-04-02T03:17:23.145 INFO:teuthology.orchestra.run.smithi029.stderr:+ zap_completion= 2022-04-02T03:17:23.146 INFO:teuthology.orchestra.run.smithi029.stderr:+ sleep 3s 2022-04-02T03:17:26.147 INFO:teuthology.orchestra.run.smithi029.stderr:+ '[' '' = 0 ']' 2022-04-02T03:17:26.147 INFO:teuthology.orchestra.run.smithi029.stderr:+ kubectl patch pv smithi104-lv-1 -p '{"spec":{"claimRef": null}}' 2022-04-02T03:17:26.558 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:26.553488 I | ceph-csi: Detected ceph CSI image version: "v3.5.1" 2022-04-02T03:17:26.560 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:26.553530 I | op-k8sutil: CSI_FORCE_CEPHFS_KERNEL_CLIENT="true" (configmap) 2022-04-02T03:17:26.560 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:26.553543 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default) 2022-04-02T03:17:26.561 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:26.553553 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default) 2022-04-02T03:17:26.561 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:26.553565 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default) 2022-04-02T03:17:26.561 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:26.553577 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default) 2022-04-02T03:17:26.561 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:26.553585 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default) 2022-04-02T03:17:26.561 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:26.553591 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default) 2022-04-02T03:17:26.562 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:26.553598 I | op-k8sutil: CSIADDONS_PORT="9070" (default) 2022-04-02T03:17:26.562 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:26.553606 I | op-k8sutil: CSIADDONS_PORT="9070" (default) 2022-04-02T03:17:26.562 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:26.553613 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default) 2022-04-02T03:17:26.562 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:26.553619 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default) 2022-04-02T03:17:26.563 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:26.553626 I | op-k8sutil: CSI_PLUGIN_PRIORITY_CLASSNAME="" (default) 2022-04-02T03:17:26.564 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:26.553634 I | op-k8sutil: CSI_PROVISIONER_PRIORITY_CLASSNAME="" (default) 2022-04-02T03:17:26.565 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:26.553644 I | op-k8sutil: CSI_ENABLE_OMAP_GENERATOR="false" (default) 2022-04-02T03:17:26.565 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:26.553657 I | op-k8sutil: CSI_ENABLE_RBD_SNAPSHOTTER="true" (configmap) 2022-04-02T03:17:26.566 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:26.553665 I | op-k8sutil: CSI_ENABLE_CEPHFS_SNAPSHOTTER="true" (configmap) 2022-04-02T03:17:26.566 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:26.553672 I | op-k8sutil: CSI_ENABLE_VOLUME_REPLICATION="false" (configmap) 2022-04-02T03:17:26.567 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:26.553679 I | op-k8sutil: CSI_ENABLE_CSIADDONS="false" (configmap) 2022-04-02T03:17:26.567 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:26.553687 I | op-k8sutil: CSI_CEPHFS_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-04-02T03:17:26.567 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:26.553695 I | op-k8sutil: CSI_RBD_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-04-02T03:17:26.568 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:26.553702 I | op-k8sutil: CSI_PLUGIN_ENABLE_SELINUX_HOST_MOUNT="false" (configmap) 2022-04-02T03:17:26.568 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:26.553708 I | ceph-csi: Kubernetes version is 1.23 2022-04-02T03:17:26.568 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:26.553717 I | op-k8sutil: ROOK_CSI_RESIZER_IMAGE="k8s.gcr.io/sig-storage/csi-resizer:v1.4.0" (default) 2022-04-02T03:17:26.568 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:26.553726 I | op-k8sutil: CSI_LOG_LEVEL="" (default) 2022-04-02T03:17:26.793 INFO:teuthology.orchestra.run.smithi029.stdout:persistentvolume/smithi104-lv-1 patched 2022-04-02T03:17:26.795 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:26.790578 I | op-k8sutil: CSI_PROVISIONER_REPLICAS="2" (configmap) 2022-04-02T03:17:26.796 INFO:teuthology.orchestra.run.smithi029.stderr:+ toolbox ceph orch apply osd --all-available-devices 2022-04-02T03:17:26.796 INFO:teuthology.orchestra.run.smithi029.stderr:+ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph orch apply osd --all-available-devices 2022-04-02T03:17:26.799 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:26.791203 I | ceph-spec: detected ceph image version: "17.0.0-11449 quincy" 2022-04-02T03:17:26.799 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:26.791234 I | ceph-cluster-controller: validating ceph version from provided image 2022-04-02T03:17:26.804 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:26.799264 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-04-02T03:17:26.804 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:26.799282 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default) 2022-04-02T03:17:26.804 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:26.799286 I | op-k8sutil: CSI_PLUGIN_TOLERATIONS="" (default) 2022-04-02T03:17:26.804 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:26.799289 I | op-k8sutil: CSI_PLUGIN_NODE_AFFINITY="" (default) 2022-04-02T03:17:26.805 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:26.799292 I | op-k8sutil: CSI_RBD_PLUGIN_TOLERATIONS="" (default) 2022-04-02T03:17:26.805 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:26.799294 I | op-k8sutil: CSI_RBD_PLUGIN_NODE_AFFINITY="" (default) 2022-04-02T03:17:26.805 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:26.799297 I | op-k8sutil: CSI_RBD_PLUGIN_RESOURCE="" (default) 2022-04-02T03:17:27.014 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:27.009760 I | op-mon: parsing mon endpoints: a=10.102.133.199:6789,b=10.107.147.59:6789,c=10.96.208.36:6789 2022-04-02T03:17:27.022 INFO:teuthology.orchestra.run.smithi029.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-04-02T03:17:27.163 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:27.158509 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-04-02T03:17:27.167 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:27.158821 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-04-02T03:17:27.209 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:27.204277 I | op-k8sutil: CSI_RBD_PROVISIONER_TOLERATIONS="" (default) 2022-04-02T03:17:27.209 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:27.204300 I | op-k8sutil: CSI_RBD_PROVISIONER_NODE_AFFINITY="" (default) 2022-04-02T03:17:27.209 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:27.204304 I | op-k8sutil: CSI_RBD_PROVISIONER_RESOURCE="" (default) 2022-04-02T03:17:27.332 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:27.327227 I | ceph-csi: successfully started CSI Ceph RBD driver 2022-04-02T03:17:27.586 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:27.582045 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-11449 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a" 2022-04-02T03:17:29.141 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:29.136627 I | op-mon: start running mons 2022-04-02T03:17:29.223 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:29.218215 I | op-k8sutil: CSI_CEPHFS_PLUGIN_TOLERATIONS="" (default) 2022-04-02T03:17:29.223 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:29.218243 I | op-k8sutil: CSI_CEPHFS_PLUGIN_NODE_AFFINITY="" (default) 2022-04-02T03:17:29.224 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:29.218252 I | op-k8sutil: CSI_CEPHFS_PLUGIN_RESOURCE="" (default) 2022-04-02T03:17:29.226 INFO:teuthology.orchestra.run.smithi029.stdout:['Success'] 2022-04-02T03:17:29.233 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:29.228443 I | ceph-cluster-controller: CR has changed for "rook-ceph". diff= v1.ClusterSpec{ 2022-04-02T03:17:29.234 INFO:tasks.rook.operator.smithi029.stdout: CephVersion: {Image: "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", AllowUnsupported: true}, 2022-04-02T03:17:29.234 INFO:tasks.rook.operator.smithi029.stdout: Storage: v1.StorageScopeSpec{ 2022-04-02T03:17:29.234 INFO:tasks.rook.operator.smithi029.stdout: ... // 3 identical fields 2022-04-02T03:17:29.234 INFO:tasks.rook.operator.smithi029.stdout: Config: nil, 2022-04-02T03:17:29.234 INFO:tasks.rook.operator.smithi029.stdout: Selection: {}, 2022-04-02T03:17:29.235 INFO:tasks.rook.operator.smithi029.stdout: StorageClassDeviceSets: []v1.StorageClassDeviceSet{ 2022-04-02T03:17:29.235 INFO:tasks.rook.operator.smithi029.stdout: ... // 9 identical elements 2022-04-02T03:17:29.235 INFO:tasks.rook.operator.smithi029.stdout: {Name: "smithi142-lv-3", Count: 1, VolumeClaimTemplates: {{ObjectMeta: {Name: "data"}, Spec: {AccessModes: {"ReadWriteOnce"}, Resources: {Requests: {s"storage": {i: {...}, s: "1", Format: "DecimalSI"}}}, VolumeName: "smithi142-lv-3", StorageClassName: &"scratch", ...}}}}, 2022-04-02T03:17:29.235 INFO:tasks.rook.operator.smithi029.stdout: {Name: "smithi142-lv-4", Count: 1, VolumeClaimTemplates: {{ObjectMeta: {Name: "data"}, Spec: {AccessModes: {"ReadWriteOnce"}, Resources: {Requests: {s"storage": {i: {...}, s: "1", Format: "DecimalSI"}}}, VolumeName: "smithi142-lv-4", StorageClassName: &"scratch", ...}}}}, 2022-04-02T03:17:29.236 INFO:tasks.rook.operator.smithi029.stdout:+ { 2022-04-02T03:17:29.237 INFO:tasks.rook.operator.smithi029.stdout:+ Name: "smithi104-lv-1", 2022-04-02T03:17:29.238 INFO:tasks.rook.operator.smithi029.stdout:+ Count: 1, 2022-04-02T03:17:29.238 INFO:tasks.rook.operator.smithi029.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{ObjectMeta: v1.ObjectMeta{...}, Spec: v1.PersistentVolumeClaimSpec{...}}}, 2022-04-02T03:17:29.238 INFO:tasks.rook.operator.smithi029.stdout:+ }, 2022-04-02T03:17:29.238 INFO:tasks.rook.operator.smithi029.stdout: }, 2022-04-02T03:17:29.238 INFO:tasks.rook.operator.smithi029.stdout: }, 2022-04-02T03:17:29.239 INFO:tasks.rook.operator.smithi029.stdout: Annotations: nil, 2022-04-02T03:17:29.239 INFO:tasks.rook.operator.smithi029.stdout: Labels: nil, 2022-04-02T03:17:29.239 INFO:tasks.rook.operator.smithi029.stdout: ... // 20 identical fields 2022-04-02T03:17:29.239 INFO:tasks.rook.operator.smithi029.stdout: } 2022-04-02T03:17:29.240 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:29.228544 I | operator: reloading operator's CRDs manager, cancelling all orchestrations! 2022-04-02T03:17:29.240 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:29.228812 E | ceph-csi: failed to reconcile failed to configure ceph csi: failed to start ceph csi drivers: failed to apply network config to cephfs plugin daemonset "csi-cephfsplugin": failed to find CephClusters: Get "https://10.96.0.1:443/apis/ceph.rook.io/v1/cephclusters": context canceled 2022-04-02T03:17:29.240 INFO:tasks.rook.operator.smithi029.stdout:I0402 03:17:29.228865 1 manager.go:148] objectbucket.io/provisioner-manager "msg"="stopping provisioner" "name"="rook-ceph.ceph.rook.io/bucket" "reason"={} 2022-04-02T03:17:29.240 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:29.228929 I | op-osd: stopping monitoring of OSDs in namespace "rook-ceph" 2022-04-02T03:17:29.240 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:29.228959 I | ceph-cluster-controller: stopping monitoring of ceph status 2022-04-02T03:17:29.241 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:29.229364 I | op-mon: stopping monitoring of mons in namespace "rook-ceph" 2022-04-02T03:17:29.251 INFO:teuthology.orchestra.run.smithi029.stderr:+ kubectl delete job rook-ceph-device-zap -n rook-ceph 2022-04-02T03:17:29.261 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:29.256300 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". failed to reconcile cluster "rook-ceph": failed to configure local ceph cluster: failed to create cluster: failed to start ceph monitors: failed to initialize ceph cluster info: failed to get cluster info: failed to get mon secrets: Get "https://10.96.0.1:443/api/v1/namespaces/rook-ceph/secrets/rook-ceph-mon": context canceled 2022-04-02T03:17:29.261 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:29.256437 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-04-02T03:17:29.261 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:29.256622 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". failed to reconcile cluster "rook-ceph": failed to load cluster info: failed to get mon secrets: context canceled 2022-04-02T03:17:29.262 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:29.256680 I | operator: successfully started the controller-runtime manager 2022-04-02T03:17:29.317 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:29.312679 I | op-k8sutil: ROOK_CURRENT_NAMESPACE_ONLY="false" (env var) 2022-04-02T03:17:29.317 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:29.312703 I | operator: watching all namespaces for Ceph CRs 2022-04-02T03:17:29.318 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:29.312736 I | operator: setting up schemes 2022-04-02T03:17:29.322 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:29.316429 I | operator: setting up the controller-runtime manager 2022-04-02T03:17:29.347 INFO:teuthology.orchestra.run.smithi029.stdout:job.batch "rook-ceph-device-zap" deleted 2022-04-02T03:17:29.352 INFO:teuthology.orchestra.run.smithi029.stderr:+ num_osd=0 2022-04-02T03:17:29.353 INFO:teuthology.orchestra.run.smithi029.stderr:+ '[' 0 '!=' 12 ']' 2022-04-02T03:17:29.353 INFO:teuthology.orchestra.run.smithi029.stderr:+ echo 'waiting for osd to come back up' 2022-04-02T03:17:29.354 INFO:teuthology.orchestra.run.smithi029.stdout:waiting for osd to come back up 2022-04-02T03:17:29.355 INFO:teuthology.orchestra.run.smithi029.stderr:++ toolbox ceph osd stat 2022-04-02T03:17:29.355 INFO:teuthology.orchestra.run.smithi029.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-04-02T03:17:29.355 INFO:teuthology.orchestra.run.smithi029.stderr:++ cut -f3 -d ' ' 2022-04-02T03:17:29.416 INFO:teuthology.orchestra.run.smithi029.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-04-02T03:17:29.524 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:29.519707 I | operator: looking for admission webhook secret "rook-ceph-admission-controller" 2022-04-02T03:17:29.527 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:29.522360 I | operator: admission webhook secret "rook-ceph-admission-controller" not found. proceeding without the admission controller 2022-04-02T03:17:29.528 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:29.522390 I | ceph-cluster-controller: successfully started 2022-04-02T03:17:29.528 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:29.522426 I | ceph-cluster-controller: enabling hotplug orchestration 2022-04-02T03:17:29.528 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:29.522439 I | ceph-crashcollector-controller: successfully started 2022-04-02T03:17:29.528 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:29.522451 I | ceph-block-pool-controller: successfully started 2022-04-02T03:17:29.529 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:29.522474 I | ceph-object-store-user-controller: successfully started 2022-04-02T03:17:29.529 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:29.522491 I | ceph-object-realm-controller: successfully started 2022-04-02T03:17:29.529 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:29.522501 I | ceph-object-zonegroup-controller: successfully started 2022-04-02T03:17:29.529 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:29.522507 I | ceph-object-zone-controller: successfully started 2022-04-02T03:17:29.530 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:29.522611 I | ceph-object-controller: successfully started 2022-04-02T03:17:29.530 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:29.522642 I | ceph-file-controller: successfully started 2022-04-02T03:17:29.530 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:29.522682 I | ceph-nfs-controller: successfully started 2022-04-02T03:17:29.530 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:29.522705 I | ceph-rbd-mirror-controller: successfully started 2022-04-02T03:17:29.531 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:29.522724 I | ceph-client-controller: successfully started 2022-04-02T03:17:29.532 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:29.522739 I | ceph-filesystem-mirror-controller: successfully started 2022-04-02T03:17:29.532 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:29.522763 I | operator: rook-ceph-operator-config-controller successfully started 2022-04-02T03:17:29.532 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:29.522784 I | ceph-csi: rook-ceph-operator-csi-controller successfully started 2022-04-02T03:17:29.532 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:29.522803 I | op-bucket-prov: rook-ceph-operator-bucket-controller successfully started 2022-04-02T03:17:29.532 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:29.522817 I | ceph-bucket-topic: successfully started 2022-04-02T03:17:29.533 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:29.522833 I | ceph-bucket-notification: successfully started 2022-04-02T03:17:29.533 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:29.522843 I | ceph-bucket-notification: successfully started 2022-04-02T03:17:29.533 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:29.522857 I | ceph-fs-subvolumegroup-controller: successfully started 2022-04-02T03:17:29.533 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:29.524118 I | operator: starting the controller-runtime manager 2022-04-02T03:17:29.632 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:29.627339 I | op-k8sutil: ROOK_CEPH_COMMANDS_TIMEOUT_SECONDS="15" (configmap) 2022-04-02T03:17:29.632 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:29.627415 I | op-k8sutil: ROOK_LOG_LEVEL="INFO" (configmap) 2022-04-02T03:17:29.633 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:29.627435 I | op-k8sutil: ROOK_ENABLE_DISCOVERY_DAEMON="false" (configmap) 2022-04-02T03:17:29.635 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:29.629149 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-04-02T03:17:29.636 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:29.630391 I | operator: rook-ceph-operator-config-controller done reconciling 2022-04-02T03:17:29.638 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:29.633505 I | op-mon: parsing mon endpoints: a=10.102.133.199:6789,b=10.107.147.59:6789,c=10.96.208.36:6789 2022-04-02T03:17:29.639 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:29.633540 I | op-k8sutil: ROOK_OBC_WATCH_OPERATOR_NAMESPACE="true" (configmap) 2022-04-02T03:17:29.640 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:29.633546 I | op-bucket-prov: ceph bucket provisioner launched watching for provisioner "rook-ceph.ceph.rook.io/bucket" 2022-04-02T03:17:29.640 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:29.634161 I | op-mon: parsing mon endpoints: a=10.102.133.199:6789,b=10.107.147.59:6789,c=10.96.208.36:6789 2022-04-02T03:17:29.640 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:29.634501 I | op-bucket-prov: successfully reconciled bucket provisioner 2022-04-02T03:17:29.641 INFO:tasks.rook.operator.smithi029.stdout:I0402 03:17:29.634542 1 manager.go:135] objectbucket.io/provisioner-manager "msg"="starting provisioner" "name"="rook-ceph.ceph.rook.io/bucket" 2022-04-02T03:17:29.648 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:29.642085 I | ceph-cluster-controller: enabling ceph mon monitoring goroutine for cluster "rook-ceph" 2022-04-02T03:17:29.648 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:29.642106 I | ceph-cluster-controller: enabling ceph osd monitoring goroutine for cluster "rook-ceph" 2022-04-02T03:17:29.648 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:29.642114 I | ceph-cluster-controller: enabling ceph status monitoring goroutine for cluster "rook-ceph" 2022-04-02T03:17:29.672 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:29.667358 I | ceph-csi: successfully created csi config map "rook-ceph-csi-config" 2022-04-02T03:17:29.675 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:29.669372 I | ceph-spec: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a... 2022-04-02T03:17:29.744 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:29.737752 I | op-k8sutil: ROOK_CSI_ENABLE_RBD="true" (configmap) 2022-04-02T03:17:29.744 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:29.737776 I | op-k8sutil: ROOK_CSI_ENABLE_CEPHFS="true" (configmap) 2022-04-02T03:17:29.744 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:29.737783 I | op-k8sutil: ROOK_CSI_ALLOW_UNSUPPORTED_VERSION="false" (configmap) 2022-04-02T03:17:29.744 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:29.737788 I | op-k8sutil: ROOK_CSI_ENABLE_GRPC_METRICS="false" (configmap) 2022-04-02T03:17:29.745 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:29.737793 I | op-k8sutil: CSI_ENABLE_HOST_NETWORK="true" (default) 2022-04-02T03:17:29.745 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:29.737799 I | op-k8sutil: ROOK_CSI_CEPH_IMAGE="quay.io/cephcsi/cephcsi:v3.5.1" (default) 2022-04-02T03:17:29.745 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:29.737806 I | op-k8sutil: ROOK_CSI_REGISTRAR_IMAGE="k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.5.0" (default) 2022-04-02T03:17:29.745 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:29.737813 I | op-k8sutil: ROOK_CSI_PROVISIONER_IMAGE="k8s.gcr.io/sig-storage/csi-provisioner:v3.1.0" (default) 2022-04-02T03:17:29.746 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:29.737819 I | op-k8sutil: ROOK_CSI_ATTACHER_IMAGE="k8s.gcr.io/sig-storage/csi-attacher:v3.4.0" (default) 2022-04-02T03:17:29.746 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:29.737825 I | op-k8sutil: ROOK_CSI_SNAPSHOTTER_IMAGE="k8s.gcr.io/sig-storage/csi-snapshotter:v5.0.1" (default) 2022-04-02T03:17:29.746 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:29.737830 I | op-k8sutil: ROOK_CSI_KUBELET_DIR_PATH="/var/lib/kubelet" (default) 2022-04-02T03:17:29.746 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:29.737836 I | op-k8sutil: CSI_VOLUME_REPLICATION_IMAGE="quay.io/csiaddons/volumereplication-operator:v0.3.0" (default) 2022-04-02T03:17:29.747 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:29.737842 I | op-k8sutil: ROOK_CSIADDONS_IMAGE="quay.io/csiaddons/k8s-sidecar:v0.2.1" (default) 2022-04-02T03:17:29.747 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:29.737846 I | op-k8sutil: ROOK_CSI_CEPHFS_POD_LABELS="" (default) 2022-04-02T03:17:29.747 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:29.737850 I | op-k8sutil: ROOK_CSI_RBD_POD_LABELS="" (default) 2022-04-02T03:17:29.747 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:29.737856 I | ceph-csi: detecting the ceph csi image version for image "quay.io/cephcsi/cephcsi:v3.5.1" 2022-04-02T03:17:29.747 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:29.737910 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-04-02T03:17:29.748 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:29.737921 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default) 2022-04-02T03:17:29.875 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:29.870389 I | op-k8sutil: Removing previous job rook-ceph-csi-detect-version to start a new one 2022-04-02T03:17:29.880 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:29.875920 I | op-k8sutil: batch job rook-ceph-csi-detect-version still exists 2022-04-02T03:17:29.908 INFO:teuthology.orchestra.run.smithi029.stderr:+ num_osd=11 2022-04-02T03:17:29.908 INFO:teuthology.orchestra.run.smithi029.stderr:+ sleep 30s 2022-04-02T03:17:32.887 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:32.882932 I | op-k8sutil: batch job rook-ceph-csi-detect-version deleted 2022-04-02T03:17:42.293 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:42.288690 I | ceph-spec: detected ceph image version: "17.0.0-11449 quincy" 2022-04-02T03:17:42.293 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:42.288718 I | ceph-cluster-controller: validating ceph version from provided image 2022-04-02T03:17:42.657 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:42.653061 I | op-mon: parsing mon endpoints: a=10.102.133.199:6789,b=10.107.147.59:6789,c=10.96.208.36:6789 2022-04-02T03:17:42.797 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:42.792979 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-04-02T03:17:42.802 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:42.793327 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-04-02T03:17:43.227 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:43.222099 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-11449 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a" 2022-04-02T03:17:44.203 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:44.198538 I | op-mon: start running mons 2022-04-02T03:17:44.616 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:44.611292 I | op-mon: parsing mon endpoints: a=10.102.133.199:6789,b=10.107.147.59:6789,c=10.96.208.36:6789 2022-04-02T03:17:45.188 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:45.183486 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.102.133.199:6789","10.107.147.59:6789","10.96.208.36:6789"]}] data:a=10.102.133.199:6789,b=10.107.147.59:6789,c=10.96.208.36:6789 mapping:{"node":{"a":{"Name":"smithi104","Hostname":"smithi104","Address":"172.21.15.104"},"b":{"Name":"smithi142","Hostname":"smithi142","Address":"172.21.15.142"},"c":{"Name":"smithi029","Hostname":"smithi029","Address":"172.21.15.29"}}} maxMonId:2] 2022-04-02T03:17:45.296 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:45.292041 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-04-02T03:17:45.301 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:45.292343 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-04-02T03:17:46.742 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:46.735889 I | ceph-csi: Detected ceph CSI image version: "v3.5.1" 2022-04-02T03:17:46.742 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:46.735931 I | op-k8sutil: CSI_FORCE_CEPHFS_KERNEL_CLIENT="true" (configmap) 2022-04-02T03:17:46.743 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:46.735943 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default) 2022-04-02T03:17:46.743 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:46.735951 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default) 2022-04-02T03:17:46.743 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:46.735959 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default) 2022-04-02T03:17:46.743 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:46.735965 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default) 2022-04-02T03:17:46.744 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:46.735971 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default) 2022-04-02T03:17:46.744 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:46.735978 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default) 2022-04-02T03:17:46.744 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:46.735988 I | op-k8sutil: CSIADDONS_PORT="9070" (default) 2022-04-02T03:17:46.744 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:46.735999 I | op-k8sutil: CSIADDONS_PORT="9070" (default) 2022-04-02T03:17:46.744 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:46.736010 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default) 2022-04-02T03:17:46.745 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:46.736026 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default) 2022-04-02T03:17:46.745 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:46.736037 I | op-k8sutil: CSI_PLUGIN_PRIORITY_CLASSNAME="" (default) 2022-04-02T03:17:46.745 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:46.736048 I | op-k8sutil: CSI_PROVISIONER_PRIORITY_CLASSNAME="" (default) 2022-04-02T03:17:46.745 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:46.736059 I | op-k8sutil: CSI_ENABLE_OMAP_GENERATOR="false" (default) 2022-04-02T03:17:46.746 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:46.736073 I | op-k8sutil: CSI_ENABLE_RBD_SNAPSHOTTER="true" (configmap) 2022-04-02T03:17:46.746 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:46.736084 I | op-k8sutil: CSI_ENABLE_CEPHFS_SNAPSHOTTER="true" (configmap) 2022-04-02T03:17:46.746 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:46.736096 I | op-k8sutil: CSI_ENABLE_VOLUME_REPLICATION="false" (configmap) 2022-04-02T03:17:46.747 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:46.736106 I | op-k8sutil: CSI_ENABLE_CSIADDONS="false" (configmap) 2022-04-02T03:17:46.747 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:46.736115 I | op-k8sutil: CSI_CEPHFS_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-04-02T03:17:46.748 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:46.736128 I | op-k8sutil: CSI_RBD_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-04-02T03:17:46.748 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:46.736140 I | op-k8sutil: CSI_PLUGIN_ENABLE_SELINUX_HOST_MOUNT="false" (configmap) 2022-04-02T03:17:46.749 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:46.736147 I | ceph-csi: Kubernetes version is 1.23 2022-04-02T03:17:46.749 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:46.736156 I | op-k8sutil: ROOK_CSI_RESIZER_IMAGE="k8s.gcr.io/sig-storage/csi-resizer:v1.4.0" (default) 2022-04-02T03:17:46.750 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:46.736166 I | op-k8sutil: CSI_LOG_LEVEL="" (default) 2022-04-02T03:17:46.834 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:46.829288 I | op-mon: targeting the mon count 3 2022-04-02T03:17:46.838 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:46.830805 I | op-k8sutil: CSI_PROVISIONER_REPLICAS="2" (configmap) 2022-04-02T03:17:46.847 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:46.842768 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-04-02T03:17:46.847 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:46.842792 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default) 2022-04-02T03:17:46.848 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:46.842797 I | op-k8sutil: CSI_PLUGIN_TOLERATIONS="" (default) 2022-04-02T03:17:46.848 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:46.842801 I | op-k8sutil: CSI_PLUGIN_NODE_AFFINITY="" (default) 2022-04-02T03:17:46.848 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:46.842805 I | op-k8sutil: CSI_RBD_PLUGIN_TOLERATIONS="" (default) 2022-04-02T03:17:46.848 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:46.842809 I | op-k8sutil: CSI_RBD_PLUGIN_NODE_AFFINITY="" (default) 2022-04-02T03:17:46.849 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:46.842815 I | op-k8sutil: CSI_RBD_PLUGIN_RESOURCE="" (default) 2022-04-02T03:17:46.959 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:46.955138 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-04-02T03:17:47.124 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:47.120003 I | op-k8sutil: CSI_RBD_PROVISIONER_TOLERATIONS="" (default) 2022-04-02T03:17:47.125 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:47.120028 I | op-k8sutil: CSI_RBD_PROVISIONER_NODE_AFFINITY="" (default) 2022-04-02T03:17:47.125 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:47.120034 I | op-k8sutil: CSI_RBD_PROVISIONER_RESOURCE="" (default) 2022-04-02T03:17:47.266 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:47.261382 I | ceph-csi: successfully started CSI Ceph RBD driver 2022-04-02T03:17:47.310 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:47.305955 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-04-02T03:17:47.311 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:47.305978 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-04-02T03:17:47.657 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:47.652185 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-04-02T03:17:47.657 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:47.652222 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-04-02T03:17:47.992 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:47.988029 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-04-02T03:17:47.993 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:47.988050 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-04-02T03:17:48.480 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:48.475354 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-04-02T03:17:48.480 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:48.475388 I | op-config: deleting "log file" option from the mon configuration database 2022-04-02T03:17:48.715 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:48.710061 I | op-k8sutil: CSI_CEPHFS_PLUGIN_TOLERATIONS="" (default) 2022-04-02T03:17:48.715 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:48.710085 I | op-k8sutil: CSI_CEPHFS_PLUGIN_NODE_AFFINITY="" (default) 2022-04-02T03:17:48.716 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:48.710092 I | op-k8sutil: CSI_CEPHFS_PLUGIN_RESOURCE="" (default) 2022-04-02T03:17:48.768 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:48.764081 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_TOLERATIONS="" (default) 2022-04-02T03:17:48.769 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:48.764109 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_NODE_AFFINITY="" (default) 2022-04-02T03:17:48.769 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:48.764118 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_RESOURCE="" (default) 2022-04-02T03:17:48.831 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:48.827036 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-04-02T03:17:48.832 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:48.827082 I | op-mon: checking for basic quorum with existing mons 2022-04-02T03:17:48.887 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:48.883416 I | ceph-csi: successfully started CSI CephFS driver 2022-04-02T03:17:49.111 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:49.106531 I | op-mon: mon "a" ip is 10.102.133.199 2022-04-02T03:17:49.116 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:49.111482 I | op-k8sutil: CSI_RBD_FSGROUPPOLICY="ReadWriteOnceWithFSType" (configmap) 2022-04-02T03:17:49.187 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:49.182593 I | ceph-csi: CSIDriver object updated for driver "rook-ceph.rbd.csi.ceph.com" 2022-04-02T03:17:49.187 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:49.182623 I | op-k8sutil: CSI_CEPHFS_FSGROUPPOLICY="ReadWriteOnceWithFSType" (configmap) 2022-04-02T03:17:49.201 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:49.197428 I | ceph-csi: CSIDriver object updated for driver "rook-ceph.cephfs.csi.ceph.com" 2022-04-02T03:17:49.420 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:49.415933 I | op-mon: mon "b" ip is 10.107.147.59 2022-04-02T03:17:50.220 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:50.215336 I | op-mon: mon "c" ip is 10.96.208.36 2022-04-02T03:17:50.832 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:50.827373 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.102.133.199:6789","10.107.147.59:6789","10.96.208.36:6789"]}] data:a=10.102.133.199:6789,b=10.107.147.59:6789,c=10.96.208.36:6789 mapping:{"node":{"a":{"Name":"smithi104","Hostname":"smithi104","Address":"172.21.15.104"},"b":{"Name":"smithi142","Hostname":"smithi142","Address":"172.21.15.142"},"c":{"Name":"smithi029","Hostname":"smithi029","Address":"172.21.15.29"}}} maxMonId:2] 2022-04-02T03:17:51.419 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:51.413927 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-04-02T03:17:51.419 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:51.414188 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-04-02T03:17:51.840 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:51.835552 I | op-mon: deployment for mon rook-ceph-mon-a already exists. updating if needed 2022-04-02T03:17:51.848 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:51.844432 I | op-k8sutil: deployment "rook-ceph-mon-a" did not change, nothing to update 2022-04-02T03:17:51.849 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:51.844460 I | op-mon: waiting for mon quorum with [a b c] 2022-04-02T03:17:52.427 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:52.422826 I | op-mon: mons running: [a b c] 2022-04-02T03:17:52.838 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:52.834094 I | op-mon: Monitors in quorum: [a b c] 2022-04-02T03:17:52.879 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:52.875314 I | op-mon: deployment for mon rook-ceph-mon-b already exists. updating if needed 2022-04-02T03:17:52.893 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:52.885182 I | op-k8sutil: deployment "rook-ceph-mon-b" did not change, nothing to update 2022-04-02T03:17:52.894 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:52.885216 I | op-mon: waiting for mon quorum with [a b c] 2022-04-02T03:17:53.026 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:53.021428 I | op-mon: mons running: [a b c] 2022-04-02T03:17:53.487 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:53.482128 I | op-mon: Monitors in quorum: [a b c] 2022-04-02T03:17:53.497 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:53.492462 I | op-mon: deployment for mon rook-ceph-mon-c already exists. updating if needed 2022-04-02T03:17:53.507 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:53.503352 I | op-k8sutil: deployment "rook-ceph-mon-c" did not change, nothing to update 2022-04-02T03:17:53.508 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:53.503379 I | op-mon: waiting for mon quorum with [a b c] 2022-04-02T03:17:53.635 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:53.631148 I | op-mon: mons running: [a b c] 2022-04-02T03:17:54.047 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:54.042320 I | op-mon: Monitors in quorum: [a b c] 2022-04-02T03:17:54.047 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:54.042346 I | op-mon: mons created: 3 2022-04-02T03:17:54.485 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:54.480853 I | op-mon: waiting for mon quorum with [a b c] 2022-04-02T03:17:54.523 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:54.519214 I | op-mon: mons running: [a b c] 2022-04-02T03:17:54.961 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:54.956048 I | op-mon: Monitors in quorum: [a b c] 2022-04-02T03:17:54.961 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:54.956094 I | cephclient: getting or creating ceph auth key "client.csi-rbd-provisioner" 2022-04-02T03:17:55.448 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:55.443377 I | cephclient: getting or creating ceph auth key "client.csi-rbd-node" 2022-04-02T03:17:55.883 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:55.878149 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-provisioner" 2022-04-02T03:17:56.309 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:56.304189 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-node" 2022-04-02T03:17:56.908 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:56.903587 I | ceph-csi: created kubernetes csi secrets for cluster "rook-ceph" 2022-04-02T03:17:56.909 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:56.903633 I | cephclient: getting or creating ceph auth key "client.crash" 2022-04-02T03:17:57.330 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:57.326241 I | ceph-crashcollector-controller: created kubernetes crash collector secret for cluster "rook-ceph" 2022-04-02T03:17:57.331 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:57.326273 I | cephclient: create rbd-mirror bootstrap peer token "client.rbd-mirror-peer" 2022-04-02T03:17:57.332 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:57.326282 I | cephclient: getting or creating ceph auth key "client.rbd-mirror-peer" 2022-04-02T03:17:57.749 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:57.741798 I | cephclient: successfully created rbd-mirror bootstrap peer token for cluster "rook-ceph" 2022-04-02T03:17:57.776 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:57.772134 I | op-mgr: start running mgr 2022-04-02T03:17:57.777 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:57.772171 I | cephclient: getting or creating ceph auth key "mgr.a" 2022-04-02T03:17:58.237 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:58.233039 I | op-mgr: deployment for mgr rook-ceph-mgr-a already exists. updating if needed 2022-04-02T03:17:58.248 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:58.243930 I | op-k8sutil: deployment "rook-ceph-mgr-a" did not change, nothing to update 2022-04-02T03:17:58.252 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:58.247599 I | op-mgr: setting services to point to mgr "a" 2022-04-02T03:17:58.854 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:58.848842 I | op-mgr: no need to update service "rook-ceph-mgr" 2022-04-02T03:17:58.854 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:58.848956 I | op-mgr: successful modules: balancer 2022-04-02T03:17:58.868 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:58.863494 I | op-osd: start running osds in namespace "rook-ceph" 2022-04-02T03:17:58.868 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:58.863531 I | op-osd: wait timeout for healthy OSDs during upgrade or restart is "10m0s" 2022-04-02T03:17:58.922 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:58.915071 I | op-osd: start provisioning the OSDs on PVCs, if needed 2022-04-02T03:17:59.030 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:59.023880 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi029-lv-1" 2022-04-02T03:17:59.030 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:59.023932 I | op-osd: OSD PVC "smithi029-lv-1-data-0xzkjz" already exists 2022-04-02T03:17:59.030 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:59.023953 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi029-lv-2" 2022-04-02T03:17:59.031 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:59.023976 I | op-osd: OSD PVC "smithi029-lv-2-data-084vgd" already exists 2022-04-02T03:17:59.031 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:59.023991 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi029-lv-3" 2022-04-02T03:17:59.031 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:59.024012 I | op-osd: OSD PVC "smithi029-lv-3-data-0f2zlj" already exists 2022-04-02T03:17:59.031 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:59.024028 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi029-lv-4" 2022-04-02T03:17:59.032 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:59.024049 I | op-osd: OSD PVC "smithi029-lv-4-data-0sxfm8" already exists 2022-04-02T03:17:59.032 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:59.024063 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi104-lv-2" 2022-04-02T03:17:59.032 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:59.024083 I | op-osd: OSD PVC "smithi104-lv-2-data-08rssq" already exists 2022-04-02T03:17:59.032 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:59.024100 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi104-lv-3" 2022-04-02T03:17:59.032 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:59.024119 I | op-osd: OSD PVC "smithi104-lv-3-data-0jlrxt" already exists 2022-04-02T03:17:59.033 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:59.024132 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi104-lv-4" 2022-04-02T03:17:59.033 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:59.024151 I | op-osd: OSD PVC "smithi104-lv-4-data-0nmqwk" already exists 2022-04-02T03:17:59.033 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:59.024166 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi142-lv-1" 2022-04-02T03:17:59.033 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:59.024188 I | op-osd: OSD PVC "smithi142-lv-1-data-08v2jt" already exists 2022-04-02T03:17:59.033 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:59.024206 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi142-lv-2" 2022-04-02T03:17:59.034 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:59.024226 I | op-osd: OSD PVC "smithi142-lv-2-data-0qdbgg" already exists 2022-04-02T03:17:59.034 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:59.024267 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi142-lv-3" 2022-04-02T03:17:59.034 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:59.024404 I | op-osd: OSD PVC "smithi142-lv-3-data-05pxj5" already exists 2022-04-02T03:17:59.034 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:59.024467 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi142-lv-4" 2022-04-02T03:17:59.035 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:59.024545 I | op-osd: OSD PVC "smithi142-lv-4-data-0hs4jx" already exists 2022-04-02T03:17:59.035 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:59.024569 I | op-osd: creating 1 new PVCs for device set "smithi104-lv-1" 2022-04-02T03:17:59.230 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:59.226194 I | op-osd: successfully provisioned PVC "smithi104-lv-1-data-0nn9k4" 2022-04-02T03:17:59.265 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:59.258419 I | op-osd: OSD will have its main bluestore block on "smithi029-lv-1-data-0xzkjz" 2022-04-02T03:17:59.265 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:59.258440 I | op-osd: skipping OSD prepare job creation for PVC "smithi029-lv-1-data-0xzkjz" because OSD daemon using the PVC already exists 2022-04-02T03:17:59.265 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:59.258445 I | op-osd: OSD will have its main bluestore block on "smithi029-lv-2-data-084vgd" 2022-04-02T03:17:59.266 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:59.258448 I | op-osd: skipping OSD prepare job creation for PVC "smithi029-lv-2-data-084vgd" because OSD daemon using the PVC already exists 2022-04-02T03:17:59.266 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:59.258452 I | op-osd: OSD will have its main bluestore block on "smithi029-lv-3-data-0f2zlj" 2022-04-02T03:17:59.266 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:59.258455 I | op-osd: skipping OSD prepare job creation for PVC "smithi029-lv-3-data-0f2zlj" because OSD daemon using the PVC already exists 2022-04-02T03:17:59.267 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:59.258458 I | op-osd: OSD will have its main bluestore block on "smithi029-lv-4-data-0sxfm8" 2022-04-02T03:17:59.267 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:59.258462 I | op-osd: skipping OSD prepare job creation for PVC "smithi029-lv-4-data-0sxfm8" because OSD daemon using the PVC already exists 2022-04-02T03:17:59.267 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:59.258465 I | op-osd: OSD will have its main bluestore block on "smithi104-lv-2-data-08rssq" 2022-04-02T03:17:59.267 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:59.258468 I | op-osd: skipping OSD prepare job creation for PVC "smithi104-lv-2-data-08rssq" because OSD daemon using the PVC already exists 2022-04-02T03:17:59.267 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:59.258472 I | op-osd: OSD will have its main bluestore block on "smithi104-lv-3-data-0jlrxt" 2022-04-02T03:17:59.268 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:59.258475 I | op-osd: skipping OSD prepare job creation for PVC "smithi104-lv-3-data-0jlrxt" because OSD daemon using the PVC already exists 2022-04-02T03:17:59.268 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:59.258478 I | op-osd: OSD will have its main bluestore block on "smithi104-lv-4-data-0nmqwk" 2022-04-02T03:17:59.268 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:59.258481 I | op-osd: skipping OSD prepare job creation for PVC "smithi104-lv-4-data-0nmqwk" because OSD daemon using the PVC already exists 2022-04-02T03:17:59.268 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:59.258485 I | op-osd: OSD will have its main bluestore block on "smithi142-lv-1-data-08v2jt" 2022-04-02T03:17:59.269 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:59.258489 I | op-osd: skipping OSD prepare job creation for PVC "smithi142-lv-1-data-08v2jt" because OSD daemon using the PVC already exists 2022-04-02T03:17:59.269 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:59.258495 I | op-osd: OSD will have its main bluestore block on "smithi142-lv-2-data-0qdbgg" 2022-04-02T03:17:59.269 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:59.258501 I | op-osd: skipping OSD prepare job creation for PVC "smithi142-lv-2-data-0qdbgg" because OSD daemon using the PVC already exists 2022-04-02T03:17:59.269 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:59.258507 I | op-osd: OSD will have its main bluestore block on "smithi142-lv-3-data-05pxj5" 2022-04-02T03:17:59.269 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:59.258513 I | op-osd: skipping OSD prepare job creation for PVC "smithi142-lv-3-data-05pxj5" because OSD daemon using the PVC already exists 2022-04-02T03:17:59.270 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:59.258519 I | op-osd: OSD will have its main bluestore block on "smithi142-lv-4-data-0hs4jx" 2022-04-02T03:17:59.270 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:59.258527 I | op-osd: skipping OSD prepare job creation for PVC "smithi142-lv-4-data-0hs4jx" because OSD daemon using the PVC already exists 2022-04-02T03:17:59.270 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:59.258535 I | op-osd: OSD will have its main bluestore block on "smithi104-lv-1-data-0nn9k4" 2022-04-02T03:17:59.695 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:59.691012 I | op-osd: started OSD provisioning job for PVC "smithi104-lv-1-data-0nn9k4" 2022-04-02T03:17:59.696 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:59.691048 I | op-osd: start provisioning the OSDs on nodes, if needed 2022-04-02T03:17:59.696 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:59.691076 I | op-osd: no nodes are defined for configuring OSDs on raw devices 2022-04-02T03:17:59.886 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:59.881727 I | op-osd: OSD orchestration status for PVC smithi104-lv-1-data-0nn9k4 is "starting" 2022-04-02T03:17:59.909 INFO:teuthology.orchestra.run.smithi029.stderr:+ '[' 11 '!=' 12 ']' 2022-04-02T03:17:59.909 INFO:teuthology.orchestra.run.smithi029.stderr:+ echo 'waiting for osd to come back up' 2022-04-02T03:17:59.910 INFO:teuthology.orchestra.run.smithi029.stdout:waiting for osd to come back up 2022-04-02T03:17:59.911 INFO:teuthology.orchestra.run.smithi029.stderr:++ toolbox ceph osd stat 2022-04-02T03:17:59.911 INFO:teuthology.orchestra.run.smithi029.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-04-02T03:17:59.911 INFO:teuthology.orchestra.run.smithi029.stderr:++ cut -f3 -d ' ' 2022-04-02T03:17:59.980 INFO:teuthology.orchestra.run.smithi029.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-04-02T03:17:59.988 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:59.983647 I | op-osd: skipping osd checks for ok-to-stop 2022-04-02T03:17:59.993 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:59.989167 I | op-osd: updating OSD 1 on PVC "smithi104-lv-2-data-08rssq" 2022-04-02T03:17:59.993 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:17:59.989198 I | op-osd: OSD will have its main bluestore block on "smithi104-lv-2-data-08rssq" 2022-04-02T03:18:00.124 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:18:00.120018 I | op-mgr: successful modules: dashboard 2022-04-02T03:18:00.126 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:18:00.120309 I | op-mgr: successful modules: mgr module(s) from the spec 2022-04-02T03:18:00.127 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:18:00.120430 I | op-mgr: successful modules: prometheus 2022-04-02T03:18:00.260 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:18:00.255858 I | op-osd: skipping osd checks for ok-to-stop 2022-04-02T03:18:00.313 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:18:00.308427 I | op-osd: updating OSD 10 on PVC "smithi029-lv-3-data-0f2zlj" 2022-04-02T03:18:00.314 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:18:00.308465 I | op-osd: OSD will have its main bluestore block on "smithi029-lv-3-data-0f2zlj" 2022-04-02T03:18:00.453 INFO:teuthology.orchestra.run.smithi029.stderr:+ num_osd=11 2022-04-02T03:18:00.454 INFO:teuthology.orchestra.run.smithi029.stderr:+ sleep 30s 2022-04-02T03:18:00.508 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:18:00.503123 I | op-osd: skipping osd checks for ok-to-stop 2022-04-02T03:18:00.515 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:18:00.509960 I | op-osd: updating OSD 11 on PVC "smithi104-lv-3-data-0jlrxt" 2022-04-02T03:18:00.515 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:18:00.509992 I | op-osd: OSD will have its main bluestore block on "smithi104-lv-3-data-0jlrxt" 2022-04-02T03:18:00.728 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:18:00.720764 I | op-osd: skipping osd checks for ok-to-stop 2022-04-02T03:18:00.732 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:18:00.728576 I | op-osd: updating OSD 2 on PVC "smithi104-lv-4-data-0nmqwk" 2022-04-02T03:18:00.733 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:18:00.728596 I | op-osd: OSD will have its main bluestore block on "smithi104-lv-4-data-0nmqwk" 2022-04-02T03:18:01.300 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:18:01.295419 I | op-osd: skipping osd checks for ok-to-stop 2022-04-02T03:18:01.395 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:18:01.391057 I | op-osd: updating OSD 3 on PVC "smithi142-lv-2-data-0qdbgg" 2022-04-02T03:18:01.396 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:18:01.391089 I | op-osd: OSD will have its main bluestore block on "smithi142-lv-2-data-0qdbgg" 2022-04-02T03:18:01.667 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:18:01.662769 I | op-mgr: successful modules: orchestrator modules 2022-04-02T03:18:02.104 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:18:02.098975 I | op-osd: skipping osd checks for ok-to-stop 2022-04-02T03:18:02.195 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:18:02.190687 I | op-osd: updating OSD 4 on PVC "smithi142-lv-3-data-05pxj5" 2022-04-02T03:18:02.195 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:18:02.190715 I | op-osd: OSD will have its main bluestore block on "smithi142-lv-3-data-05pxj5" 2022-04-02T03:18:02.901 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:18:02.897028 I | op-osd: skipping osd checks for ok-to-stop 2022-04-02T03:18:02.995 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:18:02.990897 I | op-osd: updating OSD 5 on PVC "smithi029-lv-1-data-0xzkjz" 2022-04-02T03:18:02.996 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:18:02.990937 I | op-osd: OSD will have its main bluestore block on "smithi029-lv-1-data-0xzkjz" 2022-04-02T03:18:03.719 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:18:03.714124 I | op-osd: skipping osd checks for ok-to-stop 2022-04-02T03:18:03.796 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:18:03.792124 I | op-osd: updating OSD 6 on PVC "smithi142-lv-1-data-08v2jt" 2022-04-02T03:18:03.797 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:18:03.792161 I | op-osd: OSD will have its main bluestore block on "smithi142-lv-1-data-08v2jt" 2022-04-02T03:18:04.501 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:18:04.497154 I | op-osd: skipping osd checks for ok-to-stop 2022-04-02T03:18:04.596 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:18:04.591590 I | op-osd: updating OSD 7 on PVC "smithi029-lv-2-data-084vgd" 2022-04-02T03:18:04.596 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:18:04.591629 I | op-osd: OSD will have its main bluestore block on "smithi029-lv-2-data-084vgd" 2022-04-02T03:18:05.304 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:18:05.299927 I | op-osd: skipping osd checks for ok-to-stop 2022-04-02T03:18:05.396 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:18:05.391611 I | op-osd: updating OSD 8 on PVC "smithi142-lv-4-data-0hs4jx" 2022-04-02T03:18:05.396 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:18:05.391649 I | op-osd: OSD will have its main bluestore block on "smithi142-lv-4-data-0hs4jx" 2022-04-02T03:18:06.099 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:18:06.094917 I | op-osd: skipping osd checks for ok-to-stop 2022-04-02T03:18:06.196 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:18:06.191375 I | op-osd: updating OSD 9 on PVC "smithi029-lv-4-data-0sxfm8" 2022-04-02T03:18:06.196 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:18:06.191402 I | op-osd: OSD will have its main bluestore block on "smithi029-lv-4-data-0sxfm8" 2022-04-02T03:18:15.111 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:18:15.106257 I | op-osd: OSD orchestration status for node smithi104-lv-1-data-0nn9k4 is "orchestrating" 2022-04-02T03:18:15.493 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:18:15.488761 I | op-osd: OSD orchestration status for PVC smithi104-lv-1-data-0nn9k4 is "orchestrating" 2022-04-02T03:18:21.401 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:18:21.395907 I | op-osd: OSD orchestration status for PVC smithi104-lv-1-data-0nn9k4 is "completed" 2022-04-02T03:18:21.401 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:18:21.395943 I | op-osd: creating OSD 0 on PVC "smithi104-lv-1-data-0nn9k4" 2022-04-02T03:18:21.402 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:18:21.395960 I | op-osd: OSD will have its main bluestore block on "smithi104-lv-1-data-0nn9k4" 2022-04-02T03:18:21.984 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:18:21.979999 I | op-osd: finished running OSDs in namespace "rook-ceph" 2022-04-02T03:18:21.985 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:18:21.980029 I | ceph-cluster-controller: done reconciling ceph cluster in namespace "rook-ceph" 2022-04-02T03:18:30.454 INFO:teuthology.orchestra.run.smithi029.stderr:+ '[' 11 '!=' 12 ']' 2022-04-02T03:18:30.455 INFO:teuthology.orchestra.run.smithi029.stdout:waiting for osd to come back up 2022-04-02T03:18:30.456 INFO:teuthology.orchestra.run.smithi029.stderr:+ echo 'waiting for osd to come back up' 2022-04-02T03:18:30.456 INFO:teuthology.orchestra.run.smithi029.stderr:++ toolbox ceph osd stat 2022-04-02T03:18:30.456 INFO:teuthology.orchestra.run.smithi029.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-04-02T03:18:30.456 INFO:teuthology.orchestra.run.smithi029.stderr:++ cut -f3 -d ' ' 2022-04-02T03:18:30.522 INFO:teuthology.orchestra.run.smithi029.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-04-02T03:18:30.991 INFO:teuthology.orchestra.run.smithi029.stderr:+ num_osd=11 2022-04-02T03:18:30.991 INFO:teuthology.orchestra.run.smithi029.stderr:+ sleep 30s 2022-04-02T03:19:00.992 INFO:teuthology.orchestra.run.smithi029.stderr:+ '[' 11 '!=' 12 ']' 2022-04-02T03:19:00.993 INFO:teuthology.orchestra.run.smithi029.stdout:waiting for osd to come back up 2022-04-02T03:19:00.994 INFO:teuthology.orchestra.run.smithi029.stderr:+ echo 'waiting for osd to come back up' 2022-04-02T03:19:00.994 INFO:teuthology.orchestra.run.smithi029.stderr:++ toolbox ceph osd stat 2022-04-02T03:19:00.994 INFO:teuthology.orchestra.run.smithi029.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-04-02T03:19:00.994 INFO:teuthology.orchestra.run.smithi029.stderr:++ cut -f3 -d ' ' 2022-04-02T03:19:01.073 INFO:teuthology.orchestra.run.smithi029.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-04-02T03:19:01.568 INFO:teuthology.orchestra.run.smithi029.stderr:+ num_osd=12 2022-04-02T03:19:01.569 INFO:teuthology.orchestra.run.smithi029.stderr:+ sleep 30s 2022-04-02T03:19:31.570 INFO:teuthology.orchestra.run.smithi029.stderr:+ '[' 12 '!=' 12 ']' 2022-04-02T03:19:31.572 INFO:teuthology.run_tasks:Running task rook.shell... 2022-04-02T03:19:31.586 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph orch status 2022-04-02T03:19:35.269 INFO:teuthology.orchestra.run.smithi029.stdout:Backend: rook 2022-04-02T03:19:35.270 INFO:teuthology.orchestra.run.smithi029.stdout:Available: Yes 2022-04-02T03:19:35.294 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph orch ps 2022-04-02T03:19:35.881 INFO:teuthology.orchestra.run.smithi029.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID 2022-04-02T03:19:35.882 INFO:teuthology.orchestra.run.smithi029.stdout:crashcollector.crash smithi029 running (4m) 0s ago 4m - - a1d2ec729cd2 2022-04-02T03:19:35.882 INFO:teuthology.orchestra.run.smithi029.stdout:crashcollector.crash smithi104 running (6m) 0s ago 6m - - a1d2ec729cd2 2022-04-02T03:19:35.882 INFO:teuthology.orchestra.run.smithi029.stdout:crashcollector.crash smithi142 running (6m) 0s ago 6m - - a1d2ec729cd2 2022-04-02T03:19:35.882 INFO:teuthology.orchestra.run.smithi029.stdout:mgr.a smithi029 running (6m) 0s ago 6m - - a1d2ec729cd2 2022-04-02T03:19:35.883 INFO:teuthology.orchestra.run.smithi029.stdout:mon.a smithi104 running (9m) 0s ago 9m - - a1d2ec729cd2 2022-04-02T03:19:35.883 INFO:teuthology.orchestra.run.smithi029.stdout:mon.b smithi142 running (7m) 0s ago 7m - - a1d2ec729cd2 2022-04-02T03:19:35.883 INFO:teuthology.orchestra.run.smithi029.stdout:mon.c smithi029 running (7m) 0s ago 7m - - a1d2ec729cd2 2022-04-02T03:19:35.883 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0 smithi104 running (74s) 0s ago 74s - - a1d2ec729cd2 2022-04-02T03:19:35.883 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1 smithi104 running (4m) 0s ago 4m - - a1d2ec729cd2 2022-04-02T03:19:35.884 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2 smithi104 running (4m) 0s ago 4m - - a1d2ec729cd2 2022-04-02T03:19:35.884 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3 smithi142 running (4m) 0s ago 4m - - a1d2ec729cd2 2022-04-02T03:19:35.884 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4 smithi142 running (4m) 0s ago 4m - - a1d2ec729cd2 2022-04-02T03:19:35.884 INFO:teuthology.orchestra.run.smithi029.stdout:osd.5 smithi029 running (4m) 0s ago 4m - - a1d2ec729cd2 2022-04-02T03:19:35.885 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6 smithi142 running (4m) 0s ago 4m - - a1d2ec729cd2 2022-04-02T03:19:35.885 INFO:teuthology.orchestra.run.smithi029.stdout:osd.7 smithi029 running (4m) 0s ago 4m - - a1d2ec729cd2 2022-04-02T03:19:35.885 INFO:teuthology.orchestra.run.smithi029.stdout:osd.8 smithi142 running (4m) 0s ago 4m - - a1d2ec729cd2 2022-04-02T03:19:35.885 INFO:teuthology.orchestra.run.smithi029.stdout:osd.9 smithi029 running (4m) 0s ago 4m - - a1d2ec729cd2 2022-04-02T03:19:35.885 INFO:teuthology.orchestra.run.smithi029.stdout:osd.10 smithi029 running (4m) 0s ago 4m - - a1d2ec729cd2 2022-04-02T03:19:35.886 INFO:teuthology.orchestra.run.smithi029.stdout:osd.11 smithi104 running (3m) 0s ago 3m - - a1d2ec729cd2 2022-04-02T03:19:35.900 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph orch ls 2022-04-02T03:19:36.574 INFO:teuthology.orchestra.run.smithi029.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2022-04-02T03:19:36.574 INFO:teuthology.orchestra.run.smithi029.stdout:crash 3/3 0s ago 6m * 2022-04-02T03:19:36.575 INFO:teuthology.orchestra.run.smithi029.stdout:mgr 1/1 0s ago 6m count:1 2022-04-02T03:19:36.575 INFO:teuthology.orchestra.run.smithi029.stdout:mon 3/3 0s ago 9m count:3 2022-04-02T03:19:36.576 INFO:teuthology.orchestra.run.smithi029.stdout:osd 24 0s ago 4m 2022-04-02T03:19:36.576 INFO:teuthology.orchestra.run.smithi029.stdout:osd.all-available-devices 0 0s ago - * 2022-04-02T03:19:36.599 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph orch host ls 2022-04-02T03:19:37.144 INFO:teuthology.orchestra.run.smithi029.stdout:HOST ADDR LABELS STATUS 2022-04-02T03:19:37.145 INFO:teuthology.orchestra.run.smithi029.stdout:smithi029 172.21.15.29/smithi029 2022-04-02T03:19:37.145 INFO:teuthology.orchestra.run.smithi029.stdout:smithi104 172.21.15.104/smithi104 2022-04-02T03:19:37.145 INFO:teuthology.orchestra.run.smithi029.stdout:smithi142 172.21.15.142/smithi142 2022-04-02T03:19:37.145 INFO:teuthology.orchestra.run.smithi029.stdout:3 hosts in cluster 2022-04-02T03:19:37.166 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph orch device ls 2022-04-02T03:19:37.773 INFO:teuthology.orchestra.run.smithi029.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2022-04-02T03:19:37.773 INFO:teuthology.orchestra.run.smithi029.stdout:smithi029 /dev/vg_nvme/lv_1 unknown None 107G 0s ago 2022-04-02T03:19:37.773 INFO:teuthology.orchestra.run.smithi029.stdout:smithi029 /dev/vg_nvme/lv_2 unknown None 107G 0s ago 2022-04-02T03:19:37.774 INFO:teuthology.orchestra.run.smithi029.stdout:smithi029 /dev/vg_nvme/lv_3 unknown None 107G 0s ago 2022-04-02T03:19:37.774 INFO:teuthology.orchestra.run.smithi029.stdout:smithi029 /dev/vg_nvme/lv_4 unknown None 107G 0s ago 2022-04-02T03:19:37.774 INFO:teuthology.orchestra.run.smithi029.stdout:smithi104 /dev/vg_nvme/lv_1 unknown None 107G 0s ago 2022-04-02T03:19:37.774 INFO:teuthology.orchestra.run.smithi029.stdout:smithi104 /dev/vg_nvme/lv_2 unknown None 107G 0s ago 2022-04-02T03:19:37.775 INFO:teuthology.orchestra.run.smithi029.stdout:smithi104 /dev/vg_nvme/lv_3 unknown None 107G 0s ago 2022-04-02T03:19:37.775 INFO:teuthology.orchestra.run.smithi029.stdout:smithi104 /dev/vg_nvme/lv_4 unknown None 107G 0s ago 2022-04-02T03:19:37.775 INFO:teuthology.orchestra.run.smithi029.stdout:smithi142 /dev/vg_nvme/lv_1 unknown None 107G 0s ago 2022-04-02T03:19:37.775 INFO:teuthology.orchestra.run.smithi029.stdout:smithi142 /dev/vg_nvme/lv_2 unknown None 107G 0s ago 2022-04-02T03:19:37.775 INFO:teuthology.orchestra.run.smithi029.stdout:smithi142 /dev/vg_nvme/lv_3 unknown None 107G 0s ago 2022-04-02T03:19:37.776 INFO:teuthology.orchestra.run.smithi029.stdout:smithi142 /dev/vg_nvme/lv_4 unknown None 107G 0s ago 2022-04-02T03:19:37.792 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph orch apply rgw foo 2022-04-02T03:19:38.389 INFO:teuthology.orchestra.run.smithi029.stdout:Created 2022-04-02T03:19:38.397 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:19:38.390064 I | ceph-spec: adding finalizer "cephobjectstore.ceph.rook.io" on "foo" 2022-04-02T03:19:38.415 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph orch apply mds foo 2022-04-02T03:19:38.463 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:19:38.458767 I | op-mon: parsing mon endpoints: a=10.102.133.199:6789,b=10.107.147.59:6789,c=10.96.208.36:6789 2022-04-02T03:19:38.463 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:19:38.458836 I | ceph-spec: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a... 2022-04-02T03:19:39.249 INFO:teuthology.orchestra.run.smithi029.stdout:Created 2022-04-02T03:19:39.251 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:19:39.247462 I | ceph-spec: adding finalizer "cephfilesystem.ceph.rook.io" on "foo" 2022-04-02T03:19:39.273 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph orch apply rbd-mirror 2022-04-02T03:19:39.982 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:19:39.977534 W | ceph-file-controller: failed to set filesystem "foo" status to "". failed to update object "rook-ceph/foo" status: Operation cannot be fulfilled on cephfilesystems.ceph.rook.io "foo": the object has been modified; please apply your changes to the latest version and try again 2022-04-02T03:19:40.040 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:19:40.036611 I | op-mon: parsing mon endpoints: a=10.102.133.199:6789,b=10.107.147.59:6789,c=10.96.208.36:6789 2022-04-02T03:19:40.041 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:19:40.036690 I | ceph-spec: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a... 2022-04-02T03:19:40.959 INFO:teuthology.orchestra.run.smithi029.stdout:Success 2022-04-02T03:19:40.981 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph orch apply nfs foo --port 12777 2022-04-02T03:19:41.093 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:19:41.088837 I | op-mon: parsing mon endpoints: a=10.102.133.199:6789,b=10.107.147.59:6789,c=10.96.208.36:6789 2022-04-02T03:19:41.093 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:19:41.088912 I | ceph-spec: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a... 2022-04-02T03:19:47.575 INFO:teuthology.orchestra.run.smithi029.stdout:Unable to create NFS daemon, check logs for more traceback 2022-04-02T03:19:47.575 INFO:teuthology.orchestra.run.smithi029.stdout:'RookOrchestrator' object has no attribute 'open_ioctx' 2022-04-02T03:19:47.603 DEBUG:teuthology.run_tasks:Unwinding manager rook 2022-04-02T03:19:47.616 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph orch ls -f json 2022-04-02T03:19:48.641 INFO:teuthology.orchestra.run.smithi029.stdout: 2022-04-02T03:19:48.642 INFO:teuthology.orchestra.run.smithi029.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:21.000000Z", "last_refresh": "2022-04-02T03:19:48.115737Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "mds.foo", "service_type": "mds", "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "last_refresh": "2022-04-02T03:19:48.115737Z", "running": 0, "size": 2}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:20.000000Z", "last_refresh": "2022-04-02T03:19:48.115737Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:09:56.000000Z", "last_refresh": "2022-04-02T03:19:48.115737Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:15:08.000000Z", "last_refresh": "2022-04-02T03:19:48.115737Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-02T03:19:48.115737Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_name": "rbd-mirror", "service_type": "rbd-mirror", "status": {"last_refresh": "2022-04-02T03:19:48.115737Z", "running": 0, "size": 1}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "last_refresh": "2022-04-02T03:19:48.115737Z", "running": 0, "size": 1}}] 2022-04-02T03:19:48.666 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph orch rm mds.foo 2022-04-02T03:19:49.703 INFO:teuthology.orchestra.run.smithi029.stdout:Removed cephfilesystems/foo 2022-04-02T03:19:49.727 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph orch rm rbd-mirror 2022-04-02T03:19:50.533 INFO:teuthology.orchestra.run.smithi029.stdout:Removed cephrbdmirrors/default-rbd-mirror 2022-04-02T03:19:50.554 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph orch rm rgw.foo 2022-04-02T03:19:52.447 INFO:teuthology.orchestra.run.smithi029.stdout:Removed cephobjectstores/foo 2022-04-02T03:19:52.476 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph orch ls -f json 2022-04-02T03:19:54.617 INFO:teuthology.orchestra.run.smithi029.stdout: 2022-04-02T03:19:54.618 INFO:teuthology.orchestra.run.smithi029.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:21.000000Z", "last_refresh": "2022-04-02T03:19:53.852920Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "mds.foo", "service_type": "mds", "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "last_refresh": "2022-04-02T03:19:53.852920Z", "running": 0, "size": 2}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:20.000000Z", "last_refresh": "2022-04-02T03:19:53.852920Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:09:56.000000Z", "last_refresh": "2022-04-02T03:19:53.852920Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:15:08.000000Z", "last_refresh": "2022-04-02T03:19:53.852920Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-02T03:19:53.852920Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "last_refresh": "2022-04-02T03:19:53.852920Z", "running": 0, "size": 1}}] 2022-04-02T03:19:55.955 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:19:55.950671 I | ceph-spec: detected ceph image version: "17.0.0-11449 quincy" 2022-04-02T03:19:57.346 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:19:57.342054 I | ceph-spec: detected ceph image version: "17.0.0-11449 quincy" 2022-04-02T03:19:58.242 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:19:58.237619 W | ceph-rbd-mirror-controller: (DEPRECATED) use of peer secret names in CephRBDMirror is deprecated. Please use CephBlockPool CR to configure peer secret names and import peers. 2022-04-02T03:19:58.242 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:19:58.237644 I | ceph-rbd-mirror-controller: configure rbd-mirroring with 1 workers 2022-04-02T03:19:58.243 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:19:58.237653 I | cephclient: getting or creating ceph auth key "client.rbd-mirror.a" 2022-04-02T03:19:58.660 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:19:58.655676 I | ceph-object-controller: reconciling object store deployments 2022-04-02T03:19:59.119 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:19:59.115072 I | ceph-object-controller: ceph object store gateway service running at 10.97.154.216 2022-04-02T03:19:59.120 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:19:59.115110 I | ceph-object-controller: reconciling object store pools 2022-04-02T03:19:59.515 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:19:59.511681 W | ceph-object-controller: failed to adjust the PG count for rgw metadata pools. using the general default. failed to get config setting "rgw_rados_pool_pg_num_min" for user "mon.": exit status 2 2022-04-02T03:19:59.589 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:19:59.584548 I | ceph-rbd-mirror-controller: "rook-ceph-rbd-mirror-a" deployment started 2022-04-02T03:20:00.839 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:20:00.834531 I | ceph-spec: object "rook-ceph-rbd-mirror-a" matched on delete, reconciling 2022-04-02T03:20:02.127 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:20:02.120383 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi029": the object has been modified; please apply your changes to the latest version and try again 2022-04-02T03:20:02.454 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:20:02.449603 I | ceph-spec: detected ceph image version: "17.0.0-11449 quincy" 2022-04-02T03:20:03.891 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:20:03.886842 I | ceph-file-controller: start running mdses for filesystem "foo" 2022-04-02T03:20:04.289 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:20:04.284482 I | cephclient: getting or creating ceph auth key "mds.foo-a" 2022-04-02T03:20:04.644 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph orch ls -f json 2022-04-02T03:20:05.704 INFO:teuthology.orchestra.run.smithi029.stdout: 2022-04-02T03:20:05.704 INFO:teuthology.orchestra.run.smithi029.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:21.000000Z", "last_refresh": "2022-04-02T03:20:05.328147Z", "running": 4, "size": 3}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "mds.foo", "service_type": "mds", "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "last_refresh": "2022-04-02T03:20:05.328147Z", "running": 0, "size": 2}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:20.000000Z", "last_refresh": "2022-04-02T03:20:05.328147Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:09:56.000000Z", "last_refresh": "2022-04-02T03:20:05.328147Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:15:08.000000Z", "last_refresh": "2022-04-02T03:20:05.328147Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-02T03:20:05.328147Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "last_refresh": "2022-04-02T03:20:05.328147Z", "running": 0, "size": 1}}] 2022-04-02T03:20:07.139 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:20:07.134702 I | op-mds: setting mds config flags 2022-04-02T03:20:07.139 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:20:07.134734 I | op-config: setting "mds.foo-a"="mds_join_fs"="foo" option to the mon configuration database 2022-04-02T03:20:07.956 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:20:07.951272 I | cephclient: reconciling replicated pool foo.rgw.buckets.non-ec succeeded 2022-04-02T03:20:07.959 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:20:07.952371 I | cephclient: reconciling replicated pool foo.rgw.log succeeded 2022-04-02T03:20:07.963 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:20:07.959737 I | cephclient: reconciling replicated pool foo.rgw.control succeeded 2022-04-02T03:20:09.281 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:20:09.276766 I | op-config: successfully set "mds.foo-a"="mds_join_fs"="foo" option to the mon configuration database 2022-04-02T03:20:09.297 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:20:09.288009 I | cephclient: getting or creating ceph auth key "mds.foo-b" 2022-04-02T03:20:09.581 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:20:09.577439 I | cephclient: reconciling replicated pool foo.rgw.meta succeeded 2022-04-02T03:20:09.585 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:20:09.577928 I | cephclient: reconciling replicated pool foo.rgw.buckets.index succeeded 2022-04-02T03:20:10.116 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:20:10.112498 I | op-mds: setting mds config flags 2022-04-02T03:20:10.117 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:20:10.112532 I | op-config: setting "mds.foo-b"="mds_join_fs"="foo" option to the mon configuration database 2022-04-02T03:20:11.220 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:20:11.215752 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool ".rgw.root": failed to tag pool ".rgw.root" for application "rook-ceph-rgw": failed to get application for pool ".rgw.root": failed to get current application for pool .rgw.root: exit status 2 2022-04-02T03:20:11.268 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:20:11.264409 I | op-mon: parsing mon endpoints: a=10.102.133.199:6789,b=10.107.147.59:6789,c=10.96.208.36:6789 2022-04-02T03:20:12.237 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:20:12.232745 I | op-config: successfully set "mds.foo-b"="mds_join_fs"="foo" option to the mon configuration database 2022-04-02T03:20:13.625 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:20:13.621194 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi142": the object has been modified; please apply your changes to the latest version and try again 2022-04-02T03:20:14.835 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:20:14.831153 I | ceph-file-controller: creating filesystem "foo" 2022-04-02T03:20:15.728 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph orch ls -f json 2022-04-02T03:20:17.230 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:20:17.225524 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-04-02T03:20:17.215+0000 7f7c112c6380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-04-02T03:20:17.230 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02T03:20:17.215+0000 7f7c112c6380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-04-02T03:20:17.231 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02T03:20:17.215+0000 7f7c112c6380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-04-02T03:20:17.231 INFO:tasks.rook.operator.smithi029.stdout:couldn't init storage provider. : exit status 5 2022-04-02T03:20:17.427 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:20:17.422289 I | op-mon: parsing mon endpoints: a=10.102.133.199:6789,b=10.107.147.59:6789,c=10.96.208.36:6789 2022-04-02T03:20:17.950 INFO:teuthology.orchestra.run.smithi029.stdout: 2022-04-02T03:20:17.951 INFO:teuthology.orchestra.run.smithi029.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:21.000000Z", "last_refresh": "2022-04-02T03:20:17.030320Z", "running": 5, "size": 3}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "mds.foo", "service_type": "mds", "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "last_refresh": "2022-04-02T03:20:17.030320Z", "running": 0, "size": 2}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:20.000000Z", "last_refresh": "2022-04-02T03:20:17.030320Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:09:56.000000Z", "last_refresh": "2022-04-02T03:20:17.030320Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:15:08.000000Z", "last_refresh": "2022-04-02T03:20:17.030320Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-02T03:20:17.030320Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "last_refresh": "2022-04-02T03:20:17.030320Z", "running": 0, "size": 1}}] 2022-04-02T03:20:18.259 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:20:18.254215 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-04-02T03:20:18.247+0000 7f603834d380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-04-02T03:20:18.259 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02T03:20:18.247+0000 7f603834d380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-04-02T03:20:18.259 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02T03:20:18.247+0000 7f603834d380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-04-02T03:20:18.260 INFO:tasks.rook.operator.smithi029.stdout:couldn't init storage provider. : exit status 5 2022-04-02T03:20:18.263 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:20:18.259938 I | op-mon: parsing mon endpoints: a=10.102.133.199:6789,b=10.107.147.59:6789,c=10.96.208.36:6789 2022-04-02T03:20:18.831 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:20:18.827150 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-04-02T03:20:18.819+0000 7f8ac6e8a380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-04-02T03:20:18.832 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02T03:20:18.819+0000 7f8ac6e8a380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-04-02T03:20:18.832 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02T03:20:18.819+0000 7f8ac6e8a380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-04-02T03:20:18.832 INFO:tasks.rook.operator.smithi029.stdout:couldn't init storage provider. : exit status 5 2022-04-02T03:20:19.176 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:20:19.171484 I | op-mon: parsing mon endpoints: a=10.102.133.199:6789,b=10.107.147.59:6789,c=10.96.208.36:6789 2022-04-02T03:20:19.707 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:20:19.702400 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-04-02T03:20:19.691+0000 7fa4e42b3380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-04-02T03:20:19.707 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02T03:20:19.691+0000 7fa4e42b3380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-04-02T03:20:19.707 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02T03:20:19.691+0000 7fa4e42b3380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-04-02T03:20:19.708 INFO:tasks.rook.operator.smithi029.stdout:couldn't init storage provider. : exit status 5 2022-04-02T03:20:20.074 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:20:20.068186 I | op-mon: parsing mon endpoints: a=10.102.133.199:6789,b=10.107.147.59:6789,c=10.96.208.36:6789 2022-04-02T03:20:20.801 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:20:20.797078 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-04-02T03:20:20.787+0000 7fcc4096c380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-04-02T03:20:20.801 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02T03:20:20.787+0000 7fcc4096c380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-04-02T03:20:20.802 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02T03:20:20.787+0000 7fcc4096c380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-04-02T03:20:20.802 INFO:tasks.rook.operator.smithi029.stdout:couldn't init storage provider. : exit status 5 2022-04-02T03:20:21.143 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:20:21.138620 I | op-mon: parsing mon endpoints: a=10.102.133.199:6789,b=10.107.147.59:6789,c=10.96.208.36:6789 2022-04-02T03:20:21.234 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:20:21.229767 I | cephclient: reconciling replicated pool foo-metadata succeeded 2022-04-02T03:20:21.878 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:20:21.873858 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-04-02T03:20:21.863+0000 7feddc698380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-04-02T03:20:21.879 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02T03:20:21.863+0000 7feddc698380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-04-02T03:20:21.879 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02T03:20:21.863+0000 7feddc698380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-04-02T03:20:21.879 INFO:tasks.rook.operator.smithi029.stdout:couldn't init storage provider. : exit status 5 2022-04-02T03:20:22.443 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:20:22.438810 I | op-mon: parsing mon endpoints: a=10.102.133.199:6789,b=10.107.147.59:6789,c=10.96.208.36:6789 2022-04-02T03:20:23.123 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:20:23.119146 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-04-02T03:20:23.111+0000 7eff6ba07380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-04-02T03:20:23.124 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02T03:20:23.111+0000 7eff6ba07380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-04-02T03:20:23.124 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02T03:20:23.111+0000 7eff6ba07380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-04-02T03:20:23.124 INFO:tasks.rook.operator.smithi029.stdout:couldn't init storage provider. : exit status 5 2022-04-02T03:20:23.569 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:20:23.565634 I | op-mon: parsing mon endpoints: a=10.102.133.199:6789,b=10.107.147.59:6789,c=10.96.208.36:6789 2022-04-02T03:20:24.163 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:20:24.158540 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-04-02T03:20:24.151+0000 7fc9264ee380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-04-02T03:20:24.163 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02T03:20:24.151+0000 7fc9264ee380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-04-02T03:20:24.163 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02T03:20:24.151+0000 7fc9264ee380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-04-02T03:20:24.164 INFO:tasks.rook.operator.smithi029.stdout:couldn't init storage provider. : exit status 5 2022-04-02T03:20:24.212 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:20:24.208048 I | op-mon: parsing mon endpoints: a=10.102.133.199:6789,b=10.107.147.59:6789,c=10.96.208.36:6789 2022-04-02T03:20:25.710 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:20:25.705501 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-04-02T03:20:25.694+0000 7fbb20f3d380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-04-02T03:20:25.710 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02T03:20:25.694+0000 7fbb20f3d380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-04-02T03:20:25.711 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02T03:20:25.694+0000 7fbb20f3d380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-04-02T03:20:25.711 INFO:tasks.rook.operator.smithi029.stdout:couldn't init storage provider. : exit status 5 2022-04-02T03:20:25.779 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:20:25.773530 I | op-mon: parsing mon endpoints: a=10.102.133.199:6789,b=10.107.147.59:6789,c=10.96.208.36:6789 2022-04-02T03:20:26.339 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:20:26.334129 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-04-02T03:20:26.322+0000 7fd25564e380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-04-02T03:20:26.339 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02T03:20:26.322+0000 7fd25564e380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-04-02T03:20:26.339 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02T03:20:26.326+0000 7fd25564e380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-04-02T03:20:26.340 INFO:tasks.rook.operator.smithi029.stdout:couldn't init storage provider. : exit status 5 2022-04-02T03:20:26.805 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:20:26.801123 I | cephclient: reconciling replicated pool foo-data0 succeeded 2022-04-02T03:20:27.484 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:20:27.480142 I | cephclient: creating filesystem "foo" with metadata pool "foo-metadata" and data pools [foo-data0] 2022-04-02T03:20:27.970 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph orch ls -f json 2022-04-02T03:20:28.282 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:20:28.277386 I | op-mon: parsing mon endpoints: a=10.102.133.199:6789,b=10.107.147.59:6789,c=10.96.208.36:6789 2022-04-02T03:20:28.696 INFO:teuthology.orchestra.run.smithi029.stdout: 2022-04-02T03:20:28.696 INFO:teuthology.orchestra.run.smithi029.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:26.000000Z", "last_refresh": "2022-04-02T03:20:28.492335Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "mds.foo", "service_type": "mds", "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "last_refresh": "2022-04-02T03:20:28.492335Z", "running": 0, "size": 2}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:20.000000Z", "last_refresh": "2022-04-02T03:20:28.492335Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:09:56.000000Z", "last_refresh": "2022-04-02T03:20:28.492335Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:15:08.000000Z", "last_refresh": "2022-04-02T03:20:28.492335Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-02T03:20:28.492335Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "last_refresh": "2022-04-02T03:20:28.492335Z", "running": 0, "size": 1}}] 2022-04-02T03:20:28.842 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:20:28.837708 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-04-02T03:20:28.826+0000 7fe9109b6380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-04-02T03:20:28.843 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02T03:20:28.826+0000 7fe9109b6380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-04-02T03:20:28.843 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02T03:20:28.826+0000 7fe9109b6380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-04-02T03:20:28.843 INFO:tasks.rook.operator.smithi029.stdout:couldn't init storage provider. : exit status 5 2022-04-02T03:20:29.584 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:20:29.579911 I | ceph-file-controller: created filesystem "foo" on 1 data pool(s) and metadata pool "foo-metadata" 2022-04-02T03:20:29.584 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:20:29.579941 I | cephclient: setting allow_standby_replay for filesystem "foo" 2022-04-02T03:20:30.785 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:20:30.780730 W | ceph-file-controller: failed to set filesystem "foo" status to "". failed to update object "rook-ceph/foo" status: Operation cannot be fulfilled on cephfilesystems.ceph.rook.io "foo": the object has been modified; please apply your changes to the latest version and try again 2022-04-02T03:20:30.790 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:20:30.784879 I | op-mon: parsing mon endpoints: a=10.102.133.199:6789,b=10.107.147.59:6789,c=10.96.208.36:6789 2022-04-02T03:20:30.794 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:20:30.786715 I | ceph-file-controller: CephFilesystem "rook-ceph/foo" can be deleted safely. deleting CephFilesystem "rook-ceph/foo" 2022-04-02T03:20:31.580 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:20:31.576185 I | op-config: deleting "mds_join_fs" option from the mon configuration database 2022-04-02T03:20:32.254 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:20:32.249342 I | op-config: successfully deleted "mds_join_fs" option from the mon configuration database 2022-04-02T03:20:32.254 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:20:32.249376 I | op-mds: successfully deleted mds config for "mds.foo-a" in mon configuration database 2022-04-02T03:20:32.254 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:20:32.249387 I | cephclient: deleting ceph auth "mds.foo-a" 2022-04-02T03:20:33.012 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:20:33.007982 I | op-mds: successfully deleted mds CephX key for "mds.foo-a" 2022-04-02T03:20:33.405 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:20:33.401221 I | op-config: deleting "mds_join_fs" option from the mon configuration database 2022-04-02T03:20:33.850 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:20:33.845902 I | op-config: successfully deleted "mds_join_fs" option from the mon configuration database 2022-04-02T03:20:33.851 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:20:33.845943 I | op-mds: successfully deleted mds config for "mds.foo-b" in mon configuration database 2022-04-02T03:20:33.851 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:20:33.845954 I | cephclient: deleting ceph auth "mds.foo-b" 2022-04-02T03:20:34.451 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:20:34.446421 I | op-mds: successfully deleted mds CephX key for "mds.foo-b" 2022-04-02T03:20:34.451 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:20:34.446461 I | ceph-file-controller: downing filesystem "foo" 2022-04-02T03:20:35.291 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:20:35.287332 I | ceph-file-controller: downed filesystem "foo" 2022-04-02T03:20:38.451 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:20:38.447279 I | cephclient: no images/snapshosts present in pool "foo-metadata" 2022-04-02T03:20:38.452 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:20:38.447308 I | cephclient: purging pool "foo-metadata" (id=9) 2022-04-02T03:20:38.720 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph orch ls -f json 2022-04-02T03:20:39.090 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:20:39.086436 I | op-mon: parsing mon endpoints: a=10.102.133.199:6789,b=10.107.147.59:6789,c=10.96.208.36:6789 2022-04-02T03:20:39.865 INFO:teuthology.orchestra.run.smithi029.stdout: 2022-04-02T03:20:39.865 INFO:teuthology.orchestra.run.smithi029.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:26.000000Z", "last_refresh": "2022-04-02T03:20:39.272601Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "mds.foo", "service_type": "mds", "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "last_refresh": "2022-04-02T03:20:39.272601Z", "running": 0, "size": 2}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:20.000000Z", "last_refresh": "2022-04-02T03:20:39.272601Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:09:56.000000Z", "last_refresh": "2022-04-02T03:20:39.272601Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:15:08.000000Z", "last_refresh": "2022-04-02T03:20:39.272601Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-02T03:20:39.272601Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "last_refresh": "2022-04-02T03:20:39.272601Z", "running": 0, "size": 1}}] 2022-04-02T03:20:39.906 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:20:39.901493 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-04-02T03:20:39.890+0000 7f88fc2b1380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-04-02T03:20:39.906 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02T03:20:39.890+0000 7f88fc2b1380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-04-02T03:20:39.906 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02T03:20:39.890+0000 7f88fc2b1380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-04-02T03:20:39.907 INFO:tasks.rook.operator.smithi029.stdout:couldn't init storage provider. : exit status 5 2022-04-02T03:20:41.226 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:20:41.221969 I | cephclient: purge completed for pool "foo-metadata" 2022-04-02T03:20:41.627 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:20:41.623375 I | cephclient: no images/snapshosts present in pool "foo-data0" 2022-04-02T03:20:41.628 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:20:41.623396 I | cephclient: purging pool "foo-data0" (id=10) 2022-04-02T03:20:44.672 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:20:44.667976 I | cephclient: purge completed for pool "foo-data0" 2022-04-02T03:20:44.672 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:20:44.668087 I | ceph-spec: removing finalizer "cephfilesystem.ceph.rook.io" on "foo" 2022-04-02T03:20:44.737 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:20:44.733220 I | ceph-spec: object "rook-ceph-mds-foo-a-keyring" matched on delete, reconciling 2022-04-02T03:20:44.748 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:20:44.733767 I | ceph-spec: object "rook-ceph-mds-foo-a" matched on delete, reconciling 2022-04-02T03:20:44.768 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:20:44.764441 I | ceph-spec: object "rook-ceph-mds-foo-b-keyring" matched on delete, reconciling 2022-04-02T03:20:44.772 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:20:44.764820 I | ceph-spec: object "rook-ceph-mds-foo-b" matched on delete, reconciling 2022-04-02T03:20:48.775 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:20:48.770701 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi142": the object has been modified; please apply your changes to the latest version and try again 2022-04-02T03:20:48.998 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:20:48.994141 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi142": the object has been modified; please apply your changes to the latest version and try again 2022-04-02T03:20:49.889 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph orch ls -f json 2022-04-02T03:20:50.572 INFO:teuthology.orchestra.run.smithi029.stdout: 2022-04-02T03:20:50.573 INFO:teuthology.orchestra.run.smithi029.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:26.000000Z", "last_refresh": "2022-04-02T03:20:50.400287Z", "running": 4, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:20.000000Z", "last_refresh": "2022-04-02T03:20:50.400287Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:09:56.000000Z", "last_refresh": "2022-04-02T03:20:50.400287Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:15:08.000000Z", "last_refresh": "2022-04-02T03:20:50.400287Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-02T03:20:50.400287Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "last_refresh": "2022-04-02T03:20:50.400287Z", "running": 0, "size": 1}}] 2022-04-02T03:21:00.399 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:21:00.394891 I | op-mon: parsing mon endpoints: a=10.102.133.199:6789,b=10.107.147.59:6789,c=10.96.208.36:6789 2022-04-02T03:21:00.602 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph orch ls -f json 2022-04-02T03:21:00.979 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:21:00.974538 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-04-02T03:21:00.966+0000 7f65fef56380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-04-02T03:21:00.980 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02T03:21:00.966+0000 7f65fef56380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-04-02T03:21:00.980 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02T03:21:00.966+0000 7f65fef56380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-04-02T03:21:00.980 INFO:tasks.rook.operator.smithi029.stdout:couldn't init storage provider. : exit status 5 2022-04-02T03:21:01.433 INFO:teuthology.orchestra.run.smithi029.stdout: 2022-04-02T03:21:01.433 INFO:teuthology.orchestra.run.smithi029.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:26.000000Z", "last_refresh": "2022-04-02T03:21:01.225014Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:20.000000Z", "last_refresh": "2022-04-02T03:21:01.225014Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:09:56.000000Z", "last_refresh": "2022-04-02T03:21:01.225014Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:15:08.000000Z", "last_refresh": "2022-04-02T03:21:01.225014Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-02T03:21:01.225014Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "last_refresh": "2022-04-02T03:21:01.225014Z", "running": 0, "size": 1}}] 2022-04-02T03:21:11.454 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph orch ls -f json 2022-04-02T03:21:12.122 INFO:teuthology.orchestra.run.smithi029.stdout: 2022-04-02T03:21:12.122 INFO:teuthology.orchestra.run.smithi029.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:26.000000Z", "last_refresh": "2022-04-02T03:21:11.964302Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:20.000000Z", "last_refresh": "2022-04-02T03:21:11.964302Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:09:56.000000Z", "last_refresh": "2022-04-02T03:21:11.964302Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:15:08.000000Z", "last_refresh": "2022-04-02T03:21:11.964302Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-02T03:21:11.964302Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "last_refresh": "2022-04-02T03:21:11.964302Z", "running": 0, "size": 1}}] 2022-04-02T03:21:22.149 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph orch ls -f json 2022-04-02T03:21:22.826 INFO:teuthology.orchestra.run.smithi029.stdout: 2022-04-02T03:21:22.827 INFO:teuthology.orchestra.run.smithi029.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:26.000000Z", "last_refresh": "2022-04-02T03:21:22.629387Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:20.000000Z", "last_refresh": "2022-04-02T03:21:22.629387Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:09:56.000000Z", "last_refresh": "2022-04-02T03:21:22.629387Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:15:08.000000Z", "last_refresh": "2022-04-02T03:21:22.629387Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-02T03:21:22.629387Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "last_refresh": "2022-04-02T03:21:22.629387Z", "running": 0, "size": 1}}] 2022-04-02T03:21:32.846 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph orch ls -f json 2022-04-02T03:21:33.568 INFO:teuthology.orchestra.run.smithi029.stdout: 2022-04-02T03:21:33.569 INFO:teuthology.orchestra.run.smithi029.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:26.000000Z", "last_refresh": "2022-04-02T03:21:33.370831Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:20.000000Z", "last_refresh": "2022-04-02T03:21:33.370831Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:09:56.000000Z", "last_refresh": "2022-04-02T03:21:33.370831Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:15:08.000000Z", "last_refresh": "2022-04-02T03:21:33.370831Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-02T03:21:33.370831Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "last_refresh": "2022-04-02T03:21:33.370831Z", "running": 0, "size": 1}}] 2022-04-02T03:21:41.948 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:21:41.943833 I | op-mon: parsing mon endpoints: a=10.102.133.199:6789,b=10.107.147.59:6789,c=10.96.208.36:6789 2022-04-02T03:21:42.627 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:21:42.622919 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-04-02T03:21:42.609+0000 7f17fb920380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-04-02T03:21:42.628 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02T03:21:42.609+0000 7f17fb920380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-04-02T03:21:42.628 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02T03:21:42.609+0000 7f17fb920380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-04-02T03:21:42.628 INFO:tasks.rook.operator.smithi029.stdout:couldn't init storage provider. : exit status 5 2022-04-02T03:21:43.591 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph orch ls -f json 2022-04-02T03:21:44.321 INFO:teuthology.orchestra.run.smithi029.stdout: 2022-04-02T03:21:44.322 INFO:teuthology.orchestra.run.smithi029.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:26.000000Z", "last_refresh": "2022-04-02T03:21:44.157522Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:20.000000Z", "last_refresh": "2022-04-02T03:21:44.157522Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:09:56.000000Z", "last_refresh": "2022-04-02T03:21:44.157522Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:15:08.000000Z", "last_refresh": "2022-04-02T03:21:44.157522Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-02T03:21:44.157522Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "last_refresh": "2022-04-02T03:21:44.157522Z", "running": 0, "size": 1}}] 2022-04-02T03:21:54.348 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph orch ls -f json 2022-04-02T03:21:55.067 INFO:teuthology.orchestra.run.smithi029.stdout: 2022-04-02T03:21:55.068 INFO:teuthology.orchestra.run.smithi029.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:26.000000Z", "last_refresh": "2022-04-02T03:21:54.901014Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:20.000000Z", "last_refresh": "2022-04-02T03:21:54.901014Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:09:56.000000Z", "last_refresh": "2022-04-02T03:21:54.901014Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:15:08.000000Z", "last_refresh": "2022-04-02T03:21:54.901014Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-02T03:21:54.901014Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "last_refresh": "2022-04-02T03:21:54.901014Z", "running": 0, "size": 1}}] 2022-04-02T03:22:05.092 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph orch ls -f json 2022-04-02T03:22:05.195 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:22:05.191460 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi029": the object has been modified; please apply your changes to the latest version and try again 2022-04-02T03:22:06.895 INFO:teuthology.orchestra.run.smithi029.stdout: 2022-04-02T03:22:06.895 INFO:teuthology.orchestra.run.smithi029.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:26.000000Z", "last_refresh": "2022-04-02T03:22:05.837576Z", "running": 4, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:20.000000Z", "last_refresh": "2022-04-02T03:22:05.837576Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:09:56.000000Z", "last_refresh": "2022-04-02T03:22:05.837576Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:15:08.000000Z", "last_refresh": "2022-04-02T03:22:05.837576Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-02T03:22:05.837576Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "last_refresh": "2022-04-02T03:22:05.837576Z", "running": 0, "size": 1}}] 2022-04-02T03:22:16.922 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph orch ls -f json 2022-04-02T03:22:17.585 INFO:teuthology.orchestra.run.smithi029.stdout: 2022-04-02T03:22:17.586 INFO:teuthology.orchestra.run.smithi029.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:26.000000Z", "last_refresh": "2022-04-02T03:22:17.429535Z", "running": 4, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:20.000000Z", "last_refresh": "2022-04-02T03:22:17.429535Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:09:56.000000Z", "last_refresh": "2022-04-02T03:22:17.429535Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:15:08.000000Z", "last_refresh": "2022-04-02T03:22:17.429535Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-02T03:22:17.429535Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "last_refresh": "2022-04-02T03:22:17.429535Z", "running": 0, "size": 1}}] 2022-04-02T03:22:27.607 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph orch ls -f json 2022-04-02T03:22:28.313 INFO:teuthology.orchestra.run.smithi029.stdout: 2022-04-02T03:22:28.313 INFO:teuthology.orchestra.run.smithi029.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:26.000000Z", "last_refresh": "2022-04-02T03:22:28.146333Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:20.000000Z", "last_refresh": "2022-04-02T03:22:28.146333Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:09:56.000000Z", "last_refresh": "2022-04-02T03:22:28.146333Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:15:08.000000Z", "last_refresh": "2022-04-02T03:22:28.146333Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-02T03:22:28.146333Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "last_refresh": "2022-04-02T03:22:28.146333Z", "running": 0, "size": 1}}] 2022-04-02T03:22:38.343 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph orch ls -f json 2022-04-02T03:22:39.031 INFO:teuthology.orchestra.run.smithi029.stdout: 2022-04-02T03:22:39.031 INFO:teuthology.orchestra.run.smithi029.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:26.000000Z", "last_refresh": "2022-04-02T03:22:38.845441Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:20.000000Z", "last_refresh": "2022-04-02T03:22:38.845441Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:09:56.000000Z", "last_refresh": "2022-04-02T03:22:38.845441Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:15:08.000000Z", "last_refresh": "2022-04-02T03:22:38.845441Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-02T03:22:38.845441Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "last_refresh": "2022-04-02T03:22:38.845441Z", "running": 0, "size": 1}}] 2022-04-02T03:22:49.063 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph orch ls -f json 2022-04-02T03:22:50.095 INFO:teuthology.orchestra.run.smithi029.stdout: 2022-04-02T03:22:50.096 INFO:teuthology.orchestra.run.smithi029.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:26.000000Z", "last_refresh": "2022-04-02T03:22:49.631420Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:20.000000Z", "last_refresh": "2022-04-02T03:22:49.631420Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:09:56.000000Z", "last_refresh": "2022-04-02T03:22:49.631420Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:15:08.000000Z", "last_refresh": "2022-04-02T03:22:49.631420Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-02T03:22:49.631420Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "last_refresh": "2022-04-02T03:22:49.631420Z", "running": 0, "size": 1}}] 2022-04-02T03:23:00.123 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph orch ls -f json 2022-04-02T03:23:00.835 INFO:teuthology.orchestra.run.smithi029.stdout: 2022-04-02T03:23:00.835 INFO:teuthology.orchestra.run.smithi029.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:26.000000Z", "last_refresh": "2022-04-02T03:23:00.644093Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:20.000000Z", "last_refresh": "2022-04-02T03:23:00.644093Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:09:56.000000Z", "last_refresh": "2022-04-02T03:23:00.644093Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:15:08.000000Z", "last_refresh": "2022-04-02T03:23:00.644093Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-02T03:23:00.644093Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "last_refresh": "2022-04-02T03:23:00.644093Z", "running": 0, "size": 1}}] 2022-04-02T03:23:04.552 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:23:04.549381 I | op-mon: parsing mon endpoints: a=10.102.133.199:6789,b=10.107.147.59:6789,c=10.96.208.36:6789 2022-04-02T03:23:05.252 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:23:05.248647 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-04-02T03:23:05.240+0000 7fd6e588c380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-04-02T03:23:05.253 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02T03:23:05.240+0000 7fd6e588c380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-04-02T03:23:05.253 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02T03:23:05.240+0000 7fd6e588c380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-04-02T03:23:05.253 INFO:tasks.rook.operator.smithi029.stdout:couldn't init storage provider. : exit status 5 2022-04-02T03:23:10.863 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph orch ls -f json 2022-04-02T03:23:11.625 INFO:teuthology.orchestra.run.smithi029.stdout: 2022-04-02T03:23:11.625 INFO:teuthology.orchestra.run.smithi029.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:26.000000Z", "last_refresh": "2022-04-02T03:23:11.468316Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:20.000000Z", "last_refresh": "2022-04-02T03:23:11.468316Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:09:56.000000Z", "last_refresh": "2022-04-02T03:23:11.468316Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:15:08.000000Z", "last_refresh": "2022-04-02T03:23:11.468316Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-02T03:23:11.468316Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "last_refresh": "2022-04-02T03:23:11.468316Z", "running": 0, "size": 1}}] 2022-04-02T03:23:21.652 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph orch ls -f json 2022-04-02T03:23:22.412 INFO:teuthology.orchestra.run.smithi029.stdout: 2022-04-02T03:23:22.413 INFO:teuthology.orchestra.run.smithi029.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:26.000000Z", "last_refresh": "2022-04-02T03:23:22.216752Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:20.000000Z", "last_refresh": "2022-04-02T03:23:22.216752Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:09:56.000000Z", "last_refresh": "2022-04-02T03:23:22.216752Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:15:08.000000Z", "last_refresh": "2022-04-02T03:23:22.216752Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-02T03:23:22.216752Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "last_refresh": "2022-04-02T03:23:22.216752Z", "running": 0, "size": 1}}] 2022-04-02T03:23:32.436 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph orch ls -f json 2022-04-02T03:23:33.259 INFO:teuthology.orchestra.run.smithi029.stdout: 2022-04-02T03:23:33.260 INFO:teuthology.orchestra.run.smithi029.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:26.000000Z", "last_refresh": "2022-04-02T03:23:33.044591Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:20.000000Z", "last_refresh": "2022-04-02T03:23:33.044591Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:09:56.000000Z", "last_refresh": "2022-04-02T03:23:33.044591Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:15:08.000000Z", "last_refresh": "2022-04-02T03:23:33.044591Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-02T03:23:33.044591Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "last_refresh": "2022-04-02T03:23:33.044591Z", "running": 0, "size": 1}}] 2022-04-02T03:23:43.284 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph orch ls -f json 2022-04-02T03:23:44.023 INFO:teuthology.orchestra.run.smithi029.stdout: 2022-04-02T03:23:44.024 INFO:teuthology.orchestra.run.smithi029.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:26.000000Z", "last_refresh": "2022-04-02T03:23:43.818531Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:20.000000Z", "last_refresh": "2022-04-02T03:23:43.818531Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:09:56.000000Z", "last_refresh": "2022-04-02T03:23:43.818531Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:15:08.000000Z", "last_refresh": "2022-04-02T03:23:43.818531Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-02T03:23:43.818531Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "last_refresh": "2022-04-02T03:23:43.818531Z", "running": 0, "size": 1}}] 2022-04-02T03:23:54.049 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph orch ls -f json 2022-04-02T03:23:54.716 INFO:teuthology.orchestra.run.smithi029.stdout: 2022-04-02T03:23:54.716 INFO:teuthology.orchestra.run.smithi029.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:26.000000Z", "last_refresh": "2022-04-02T03:23:54.556924Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:20.000000Z", "last_refresh": "2022-04-02T03:23:54.556924Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:09:56.000000Z", "last_refresh": "2022-04-02T03:23:54.556924Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:15:08.000000Z", "last_refresh": "2022-04-02T03:23:54.556924Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-02T03:23:54.556924Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "last_refresh": "2022-04-02T03:23:54.556924Z", "running": 0, "size": 1}}] 2022-04-02T03:24:04.740 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph orch ls -f json 2022-04-02T03:24:05.433 INFO:teuthology.orchestra.run.smithi029.stdout: 2022-04-02T03:24:05.434 INFO:teuthology.orchestra.run.smithi029.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:26.000000Z", "last_refresh": "2022-04-02T03:24:05.244245Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:20.000000Z", "last_refresh": "2022-04-02T03:24:05.244245Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:09:56.000000Z", "last_refresh": "2022-04-02T03:24:05.244245Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:15:08.000000Z", "last_refresh": "2022-04-02T03:24:05.244245Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-02T03:24:05.244245Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "last_refresh": "2022-04-02T03:24:05.244245Z", "running": 0, "size": 1}}] 2022-04-02T03:24:15.462 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph orch ls -f json 2022-04-02T03:24:16.203 INFO:teuthology.orchestra.run.smithi029.stdout: 2022-04-02T03:24:16.203 INFO:teuthology.orchestra.run.smithi029.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:26.000000Z", "last_refresh": "2022-04-02T03:24:16.037126Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:20.000000Z", "last_refresh": "2022-04-02T03:24:16.037126Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:09:56.000000Z", "last_refresh": "2022-04-02T03:24:16.037126Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:15:08.000000Z", "last_refresh": "2022-04-02T03:24:16.037126Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-02T03:24:16.037126Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "last_refresh": "2022-04-02T03:24:16.037126Z", "running": 0, "size": 1}}] 2022-04-02T03:24:26.229 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph orch ls -f json 2022-04-02T03:24:27.026 INFO:teuthology.orchestra.run.smithi029.stdout: 2022-04-02T03:24:27.026 INFO:teuthology.orchestra.run.smithi029.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:26.000000Z", "last_refresh": "2022-04-02T03:24:26.828593Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:20.000000Z", "last_refresh": "2022-04-02T03:24:26.828593Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:09:56.000000Z", "last_refresh": "2022-04-02T03:24:26.828593Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:15:08.000000Z", "last_refresh": "2022-04-02T03:24:26.828593Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-02T03:24:26.828593Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "last_refresh": "2022-04-02T03:24:26.828593Z", "running": 0, "size": 1}}] 2022-04-02T03:24:37.053 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph orch ls -f json 2022-04-02T03:24:37.735 INFO:teuthology.orchestra.run.smithi029.stdout: 2022-04-02T03:24:37.735 INFO:teuthology.orchestra.run.smithi029.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:26.000000Z", "last_refresh": "2022-04-02T03:24:37.567917Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:20.000000Z", "last_refresh": "2022-04-02T03:24:37.567917Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:09:56.000000Z", "last_refresh": "2022-04-02T03:24:37.567917Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:15:08.000000Z", "last_refresh": "2022-04-02T03:24:37.567917Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-02T03:24:37.567917Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "last_refresh": "2022-04-02T03:24:37.567917Z", "running": 0, "size": 1}}] 2022-04-02T03:24:47.760 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph orch ls -f json 2022-04-02T03:24:48.551 INFO:teuthology.orchestra.run.smithi029.stdout: 2022-04-02T03:24:48.551 INFO:teuthology.orchestra.run.smithi029.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:26.000000Z", "last_refresh": "2022-04-02T03:24:48.371901Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:20.000000Z", "last_refresh": "2022-04-02T03:24:48.371901Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:09:56.000000Z", "last_refresh": "2022-04-02T03:24:48.371901Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:15:08.000000Z", "last_refresh": "2022-04-02T03:24:48.371901Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-02T03:24:48.371901Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "last_refresh": "2022-04-02T03:24:48.371901Z", "running": 0, "size": 1}}] 2022-04-02T03:24:58.577 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph orch ls -f json 2022-04-02T03:24:59.317 INFO:teuthology.orchestra.run.smithi029.stdout: 2022-04-02T03:24:59.318 INFO:teuthology.orchestra.run.smithi029.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:26.000000Z", "last_refresh": "2022-04-02T03:24:59.092538Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:20.000000Z", "last_refresh": "2022-04-02T03:24:59.092538Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:09:56.000000Z", "last_refresh": "2022-04-02T03:24:59.092538Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:15:08.000000Z", "last_refresh": "2022-04-02T03:24:59.092538Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-02T03:24:59.092538Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "last_refresh": "2022-04-02T03:24:59.092538Z", "running": 0, "size": 1}}] 2022-04-02T03:25:09.341 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph orch ls -f json 2022-04-02T03:25:10.125 INFO:teuthology.orchestra.run.smithi029.stdout: 2022-04-02T03:25:10.125 INFO:teuthology.orchestra.run.smithi029.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:26.000000Z", "last_refresh": "2022-04-02T03:25:09.909717Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:20.000000Z", "last_refresh": "2022-04-02T03:25:09.909717Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:09:56.000000Z", "last_refresh": "2022-04-02T03:25:09.909717Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:15:08.000000Z", "last_refresh": "2022-04-02T03:25:09.909717Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-02T03:25:09.909717Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "last_refresh": "2022-04-02T03:25:09.909717Z", "running": 0, "size": 1}}] 2022-04-02T03:25:20.154 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph orch ls -f json 2022-04-02T03:25:20.862 INFO:teuthology.orchestra.run.smithi029.stdout: 2022-04-02T03:25:20.863 INFO:teuthology.orchestra.run.smithi029.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:26.000000Z", "last_refresh": "2022-04-02T03:25:20.694838Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:20.000000Z", "last_refresh": "2022-04-02T03:25:20.694838Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:09:56.000000Z", "last_refresh": "2022-04-02T03:25:20.694838Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:15:08.000000Z", "last_refresh": "2022-04-02T03:25:20.694838Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-02T03:25:20.694838Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "last_refresh": "2022-04-02T03:25:20.694838Z", "running": 0, "size": 1}}] 2022-04-02T03:25:30.889 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph orch ls -f json 2022-04-02T03:25:31.647 INFO:teuthology.orchestra.run.smithi029.stdout: 2022-04-02T03:25:31.648 INFO:teuthology.orchestra.run.smithi029.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:26.000000Z", "last_refresh": "2022-04-02T03:25:31.453314Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:20.000000Z", "last_refresh": "2022-04-02T03:25:31.453314Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:09:56.000000Z", "last_refresh": "2022-04-02T03:25:31.453314Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:15:08.000000Z", "last_refresh": "2022-04-02T03:25:31.453314Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-02T03:25:31.453314Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "last_refresh": "2022-04-02T03:25:31.453314Z", "running": 0, "size": 1}}] 2022-04-02T03:25:41.677 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph orch ls -f json 2022-04-02T03:25:42.403 INFO:teuthology.orchestra.run.smithi029.stdout: 2022-04-02T03:25:42.403 INFO:teuthology.orchestra.run.smithi029.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:26.000000Z", "last_refresh": "2022-04-02T03:25:42.240188Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:20.000000Z", "last_refresh": "2022-04-02T03:25:42.240188Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:09:56.000000Z", "last_refresh": "2022-04-02T03:25:42.240188Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:15:08.000000Z", "last_refresh": "2022-04-02T03:25:42.240188Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-02T03:25:42.240188Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "last_refresh": "2022-04-02T03:25:42.240188Z", "running": 0, "size": 1}}] 2022-04-02T03:25:49.105 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:25:49.103440 I | op-mon: parsing mon endpoints: a=10.102.133.199:6789,b=10.107.147.59:6789,c=10.96.208.36:6789 2022-04-02T03:25:49.752 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:25:49.749904 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-04-02T03:25:49.739+0000 7ff9ec63d380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-04-02T03:25:49.752 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02T03:25:49.739+0000 7ff9ec63d380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-04-02T03:25:49.753 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02T03:25:49.739+0000 7ff9ec63d380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-04-02T03:25:49.753 INFO:tasks.rook.operator.smithi029.stdout:couldn't init storage provider. : exit status 5 2022-04-02T03:25:52.421 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph orch ls -f json 2022-04-02T03:25:53.246 INFO:teuthology.orchestra.run.smithi029.stdout: 2022-04-02T03:25:53.246 INFO:teuthology.orchestra.run.smithi029.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:26.000000Z", "last_refresh": "2022-04-02T03:25:53.086991Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:20.000000Z", "last_refresh": "2022-04-02T03:25:53.086991Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:09:56.000000Z", "last_refresh": "2022-04-02T03:25:53.086991Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:15:08.000000Z", "last_refresh": "2022-04-02T03:25:53.086991Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-02T03:25:53.086991Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "last_refresh": "2022-04-02T03:25:53.086991Z", "running": 0, "size": 1}}] 2022-04-02T03:26:03.271 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph orch ls -f json 2022-04-02T03:26:04.006 INFO:teuthology.orchestra.run.smithi029.stdout: 2022-04-02T03:26:04.007 INFO:teuthology.orchestra.run.smithi029.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:26.000000Z", "last_refresh": "2022-04-02T03:26:03.788780Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:20.000000Z", "last_refresh": "2022-04-02T03:26:03.788780Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:09:56.000000Z", "last_refresh": "2022-04-02T03:26:03.788780Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:15:08.000000Z", "last_refresh": "2022-04-02T03:26:03.788780Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-02T03:26:03.788780Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "last_refresh": "2022-04-02T03:26:03.788780Z", "running": 0, "size": 1}}] 2022-04-02T03:26:14.032 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph orch ls -f json 2022-04-02T03:26:14.739 INFO:teuthology.orchestra.run.smithi029.stdout: 2022-04-02T03:26:14.739 INFO:teuthology.orchestra.run.smithi029.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:26.000000Z", "last_refresh": "2022-04-02T03:26:14.579496Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:20.000000Z", "last_refresh": "2022-04-02T03:26:14.579496Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:09:56.000000Z", "last_refresh": "2022-04-02T03:26:14.579496Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:15:08.000000Z", "last_refresh": "2022-04-02T03:26:14.579496Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-02T03:26:14.579496Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "last_refresh": "2022-04-02T03:26:14.579496Z", "running": 0, "size": 1}}] 2022-04-02T03:26:24.769 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph orch ls -f json 2022-04-02T03:26:25.475 INFO:teuthology.orchestra.run.smithi029.stdout: 2022-04-02T03:26:25.475 INFO:teuthology.orchestra.run.smithi029.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:26.000000Z", "last_refresh": "2022-04-02T03:26:25.295486Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:20.000000Z", "last_refresh": "2022-04-02T03:26:25.295486Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:09:56.000000Z", "last_refresh": "2022-04-02T03:26:25.295486Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:15:08.000000Z", "last_refresh": "2022-04-02T03:26:25.295486Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-02T03:26:25.295486Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "last_refresh": "2022-04-02T03:26:25.295486Z", "running": 0, "size": 1}}] 2022-04-02T03:26:35.501 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph orch ls -f json 2022-04-02T03:26:36.202 INFO:teuthology.orchestra.run.smithi029.stdout: 2022-04-02T03:26:36.203 INFO:teuthology.orchestra.run.smithi029.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:26.000000Z", "last_refresh": "2022-04-02T03:26:36.034009Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:20.000000Z", "last_refresh": "2022-04-02T03:26:36.034009Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:09:56.000000Z", "last_refresh": "2022-04-02T03:26:36.034009Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:15:08.000000Z", "last_refresh": "2022-04-02T03:26:36.034009Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-02T03:26:36.034009Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "last_refresh": "2022-04-02T03:26:36.034009Z", "running": 0, "size": 1}}] 2022-04-02T03:26:46.227 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph orch ls -f json 2022-04-02T03:26:47.052 INFO:teuthology.orchestra.run.smithi029.stdout: 2022-04-02T03:26:47.052 INFO:teuthology.orchestra.run.smithi029.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:26.000000Z", "last_refresh": "2022-04-02T03:26:46.806086Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:20.000000Z", "last_refresh": "2022-04-02T03:26:46.806086Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:09:56.000000Z", "last_refresh": "2022-04-02T03:26:46.806086Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:15:08.000000Z", "last_refresh": "2022-04-02T03:26:46.806086Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-02T03:26:46.806086Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "last_refresh": "2022-04-02T03:26:46.806086Z", "running": 0, "size": 1}}] 2022-04-02T03:26:57.072 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph orch ls -f json 2022-04-02T03:26:57.778 INFO:teuthology.orchestra.run.smithi029.stdout: 2022-04-02T03:26:57.779 INFO:teuthology.orchestra.run.smithi029.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:26.000000Z", "last_refresh": "2022-04-02T03:26:57.609998Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:20.000000Z", "last_refresh": "2022-04-02T03:26:57.609998Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:09:56.000000Z", "last_refresh": "2022-04-02T03:26:57.609998Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:15:08.000000Z", "last_refresh": "2022-04-02T03:26:57.609998Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-02T03:26:57.609998Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "last_refresh": "2022-04-02T03:26:57.609998Z", "running": 0, "size": 1}}] 2022-04-02T03:27:07.806 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph orch ls -f json 2022-04-02T03:27:08.497 INFO:teuthology.orchestra.run.smithi029.stdout: 2022-04-02T03:27:08.497 INFO:teuthology.orchestra.run.smithi029.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:26.000000Z", "last_refresh": "2022-04-02T03:27:08.323960Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:20.000000Z", "last_refresh": "2022-04-02T03:27:08.323960Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:09:56.000000Z", "last_refresh": "2022-04-02T03:27:08.323960Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:15:08.000000Z", "last_refresh": "2022-04-02T03:27:08.323960Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-02T03:27:08.323960Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "last_refresh": "2022-04-02T03:27:08.323960Z", "running": 0, "size": 1}}] 2022-04-02T03:27:18.526 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph orch ls -f json 2022-04-02T03:27:19.282 INFO:teuthology.orchestra.run.smithi029.stdout: 2022-04-02T03:27:19.283 INFO:teuthology.orchestra.run.smithi029.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:26.000000Z", "last_refresh": "2022-04-02T03:27:19.107591Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:20.000000Z", "last_refresh": "2022-04-02T03:27:19.107591Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:09:56.000000Z", "last_refresh": "2022-04-02T03:27:19.107591Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:15:08.000000Z", "last_refresh": "2022-04-02T03:27:19.107591Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-02T03:27:19.107591Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "last_refresh": "2022-04-02T03:27:19.107591Z", "running": 0, "size": 1}}] 2022-04-02T03:27:29.309 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph orch ls -f json 2022-04-02T03:27:30.041 INFO:teuthology.orchestra.run.smithi029.stdout: 2022-04-02T03:27:30.042 INFO:teuthology.orchestra.run.smithi029.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:26.000000Z", "last_refresh": "2022-04-02T03:27:29.873509Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:20.000000Z", "last_refresh": "2022-04-02T03:27:29.873509Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:09:56.000000Z", "last_refresh": "2022-04-02T03:27:29.873509Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:15:08.000000Z", "last_refresh": "2022-04-02T03:27:29.873509Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-02T03:27:29.873509Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "last_refresh": "2022-04-02T03:27:29.873509Z", "running": 0, "size": 1}}] 2022-04-02T03:27:40.068 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph orch ls -f json 2022-04-02T03:27:40.804 INFO:teuthology.orchestra.run.smithi029.stdout: 2022-04-02T03:27:40.804 INFO:teuthology.orchestra.run.smithi029.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:26.000000Z", "last_refresh": "2022-04-02T03:27:40.624611Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:20.000000Z", "last_refresh": "2022-04-02T03:27:40.624611Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:09:56.000000Z", "last_refresh": "2022-04-02T03:27:40.624611Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:15:08.000000Z", "last_refresh": "2022-04-02T03:27:40.624611Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-02T03:27:40.624611Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "last_refresh": "2022-04-02T03:27:40.624611Z", "running": 0, "size": 1}}] 2022-04-02T03:27:50.830 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph orch ls -f json 2022-04-02T03:27:51.785 INFO:teuthology.orchestra.run.smithi029.stdout: 2022-04-02T03:27:51.785 INFO:teuthology.orchestra.run.smithi029.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:26.000000Z", "last_refresh": "2022-04-02T03:27:51.333142Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:20.000000Z", "last_refresh": "2022-04-02T03:27:51.333142Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:09:56.000000Z", "last_refresh": "2022-04-02T03:27:51.333142Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:15:08.000000Z", "last_refresh": "2022-04-02T03:27:51.333142Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-02T03:27:51.333142Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "last_refresh": "2022-04-02T03:27:51.333142Z", "running": 0, "size": 1}}] 2022-04-02T03:28:01.816 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph orch ls -f json 2022-04-02T03:28:02.543 INFO:teuthology.orchestra.run.smithi029.stdout: 2022-04-02T03:28:02.544 INFO:teuthology.orchestra.run.smithi029.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:26.000000Z", "last_refresh": "2022-04-02T03:28:02.362897Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:20.000000Z", "last_refresh": "2022-04-02T03:28:02.362897Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:09:56.000000Z", "last_refresh": "2022-04-02T03:28:02.362897Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:15:08.000000Z", "last_refresh": "2022-04-02T03:28:02.362897Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-02T03:28:02.362897Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "last_refresh": "2022-04-02T03:28:02.362897Z", "running": 0, "size": 1}}] 2022-04-02T03:28:12.574 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph orch ls -f json 2022-04-02T03:28:13.263 INFO:teuthology.orchestra.run.smithi029.stdout: 2022-04-02T03:28:13.263 INFO:teuthology.orchestra.run.smithi029.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:26.000000Z", "last_refresh": "2022-04-02T03:28:13.090845Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:20.000000Z", "last_refresh": "2022-04-02T03:28:13.090845Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:09:56.000000Z", "last_refresh": "2022-04-02T03:28:13.090845Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:15:08.000000Z", "last_refresh": "2022-04-02T03:28:13.090845Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-02T03:28:13.090845Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "last_refresh": "2022-04-02T03:28:13.090845Z", "running": 0, "size": 1}}] 2022-04-02T03:28:23.291 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph orch ls -f json 2022-04-02T03:28:24.017 INFO:teuthology.orchestra.run.smithi029.stdout: 2022-04-02T03:28:24.018 INFO:teuthology.orchestra.run.smithi029.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:26.000000Z", "last_refresh": "2022-04-02T03:28:23.802590Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:20.000000Z", "last_refresh": "2022-04-02T03:28:23.802590Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:09:56.000000Z", "last_refresh": "2022-04-02T03:28:23.802590Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:15:08.000000Z", "last_refresh": "2022-04-02T03:28:23.802590Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-02T03:28:23.802590Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "last_refresh": "2022-04-02T03:28:23.802590Z", "running": 0, "size": 1}}] 2022-04-02T03:28:34.044 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph orch ls -f json 2022-04-02T03:28:34.736 INFO:teuthology.orchestra.run.smithi029.stdout: 2022-04-02T03:28:34.736 INFO:teuthology.orchestra.run.smithi029.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:26.000000Z", "last_refresh": "2022-04-02T03:28:34.572706Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:20.000000Z", "last_refresh": "2022-04-02T03:28:34.572706Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:09:56.000000Z", "last_refresh": "2022-04-02T03:28:34.572706Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:15:08.000000Z", "last_refresh": "2022-04-02T03:28:34.572706Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-02T03:28:34.572706Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "last_refresh": "2022-04-02T03:28:34.572706Z", "running": 0, "size": 1}}] 2022-04-02T03:28:44.761 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph orch ls -f json 2022-04-02T03:28:45.472 INFO:teuthology.orchestra.run.smithi029.stdout: 2022-04-02T03:28:45.472 INFO:teuthology.orchestra.run.smithi029.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:26.000000Z", "last_refresh": "2022-04-02T03:28:45.301643Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:20.000000Z", "last_refresh": "2022-04-02T03:28:45.301643Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:09:56.000000Z", "last_refresh": "2022-04-02T03:28:45.301643Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:15:08.000000Z", "last_refresh": "2022-04-02T03:28:45.301643Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-02T03:28:45.301643Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "last_refresh": "2022-04-02T03:28:45.301643Z", "running": 0, "size": 1}}] 2022-04-02T03:28:55.499 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph orch ls -f json 2022-04-02T03:28:56.239 INFO:teuthology.orchestra.run.smithi029.stdout: 2022-04-02T03:28:56.239 INFO:teuthology.orchestra.run.smithi029.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:26.000000Z", "last_refresh": "2022-04-02T03:28:56.087873Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:20.000000Z", "last_refresh": "2022-04-02T03:28:56.087873Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:09:56.000000Z", "last_refresh": "2022-04-02T03:28:56.087873Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:15:08.000000Z", "last_refresh": "2022-04-02T03:28:56.087873Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-02T03:28:56.087873Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "last_refresh": "2022-04-02T03:28:56.087873Z", "running": 0, "size": 1}}] 2022-04-02T03:29:06.264 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph orch ls -f json 2022-04-02T03:29:06.977 INFO:teuthology.orchestra.run.smithi029.stdout: 2022-04-02T03:29:06.978 INFO:teuthology.orchestra.run.smithi029.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:26.000000Z", "last_refresh": "2022-04-02T03:29:06.790481Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:20.000000Z", "last_refresh": "2022-04-02T03:29:06.790481Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:09:56.000000Z", "last_refresh": "2022-04-02T03:29:06.790481Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:15:08.000000Z", "last_refresh": "2022-04-02T03:29:06.790481Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-02T03:29:06.790481Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "last_refresh": "2022-04-02T03:29:06.790481Z", "running": 0, "size": 1}}] 2022-04-02T03:29:17.002 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph orch ls -f json 2022-04-02T03:29:17.750 INFO:teuthology.orchestra.run.smithi029.stdout: 2022-04-02T03:29:17.750 INFO:teuthology.orchestra.run.smithi029.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:26.000000Z", "last_refresh": "2022-04-02T03:29:17.552150Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:20.000000Z", "last_refresh": "2022-04-02T03:29:17.552150Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:09:56.000000Z", "last_refresh": "2022-04-02T03:29:17.552150Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:15:08.000000Z", "last_refresh": "2022-04-02T03:29:17.552150Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-02T03:29:17.552150Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "last_refresh": "2022-04-02T03:29:17.552150Z", "running": 0, "size": 1}}] 2022-04-02T03:29:27.771 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph orch ls -f json 2022-04-02T03:29:28.481 INFO:teuthology.orchestra.run.smithi029.stdout: 2022-04-02T03:29:28.481 INFO:teuthology.orchestra.run.smithi029.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:26.000000Z", "last_refresh": "2022-04-02T03:29:28.315849Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:20.000000Z", "last_refresh": "2022-04-02T03:29:28.315849Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:09:56.000000Z", "last_refresh": "2022-04-02T03:29:28.315849Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:15:08.000000Z", "last_refresh": "2022-04-02T03:29:28.315849Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-02T03:29:28.315849Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "last_refresh": "2022-04-02T03:29:28.315849Z", "running": 0, "size": 1}}] 2022-04-02T03:29:38.506 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph orch ls -f json 2022-04-02T03:29:42.160 INFO:teuthology.orchestra.run.smithi029.stdout: 2022-04-02T03:29:42.160 INFO:teuthology.orchestra.run.smithi029.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:26.000000Z", "last_refresh": "2022-04-02T03:29:41.979662Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:20.000000Z", "last_refresh": "2022-04-02T03:29:41.979662Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:09:56.000000Z", "last_refresh": "2022-04-02T03:29:41.979662Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:15:08.000000Z", "last_refresh": "2022-04-02T03:29:41.979662Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-02T03:29:41.979662Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "last_refresh": "2022-04-02T03:29:41.979662Z", "running": 0, "size": 1}}] 2022-04-02T03:29:52.186 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph orch ls -f json 2022-04-02T03:29:52.915 INFO:teuthology.orchestra.run.smithi029.stdout: 2022-04-02T03:29:52.916 INFO:teuthology.orchestra.run.smithi029.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:26.000000Z", "last_refresh": "2022-04-02T03:29:52.705819Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:20.000000Z", "last_refresh": "2022-04-02T03:29:52.705819Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:09:56.000000Z", "last_refresh": "2022-04-02T03:29:52.705819Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:15:08.000000Z", "last_refresh": "2022-04-02T03:29:52.705819Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-02T03:29:52.705819Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "last_refresh": "2022-04-02T03:29:52.705819Z", "running": 0, "size": 1}}] 2022-04-02T03:30:02.941 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph orch ls -f json 2022-04-02T03:30:03.598 INFO:teuthology.orchestra.run.smithi029.stdout: 2022-04-02T03:30:03.598 INFO:teuthology.orchestra.run.smithi029.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:26.000000Z", "last_refresh": "2022-04-02T03:30:03.442868Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:20.000000Z", "last_refresh": "2022-04-02T03:30:03.442868Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:09:56.000000Z", "last_refresh": "2022-04-02T03:30:03.442868Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:15:08.000000Z", "last_refresh": "2022-04-02T03:30:03.442868Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-02T03:30:03.442868Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "last_refresh": "2022-04-02T03:30:03.442868Z", "running": 0, "size": 1}}] 2022-04-02T03:30:13.626 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph orch ls -f json 2022-04-02T03:30:14.370 INFO:teuthology.orchestra.run.smithi029.stdout: 2022-04-02T03:30:14.371 INFO:teuthology.orchestra.run.smithi029.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:26.000000Z", "last_refresh": "2022-04-02T03:30:14.190746Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:20.000000Z", "last_refresh": "2022-04-02T03:30:14.190746Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:09:56.000000Z", "last_refresh": "2022-04-02T03:30:14.190746Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:15:08.000000Z", "last_refresh": "2022-04-02T03:30:14.190746Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-02T03:30:14.190746Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "last_refresh": "2022-04-02T03:30:14.190746Z", "running": 0, "size": 1}}] 2022-04-02T03:30:24.397 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph orch ls -f json 2022-04-02T03:30:25.104 INFO:teuthology.orchestra.run.smithi029.stdout: 2022-04-02T03:30:25.105 INFO:teuthology.orchestra.run.smithi029.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:26.000000Z", "last_refresh": "2022-04-02T03:30:24.932785Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:20.000000Z", "last_refresh": "2022-04-02T03:30:24.932785Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:09:56.000000Z", "last_refresh": "2022-04-02T03:30:24.932785Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:15:08.000000Z", "last_refresh": "2022-04-02T03:30:24.932785Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-02T03:30:24.932785Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "last_refresh": "2022-04-02T03:30:24.932785Z", "running": 0, "size": 1}}] 2022-04-02T03:30:35.130 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph orch ls -f json 2022-04-02T03:30:35.837 INFO:teuthology.orchestra.run.smithi029.stdout: 2022-04-02T03:30:35.837 INFO:teuthology.orchestra.run.smithi029.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:26.000000Z", "last_refresh": "2022-04-02T03:30:35.664763Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:20.000000Z", "last_refresh": "2022-04-02T03:30:35.664763Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:09:56.000000Z", "last_refresh": "2022-04-02T03:30:35.664763Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:15:08.000000Z", "last_refresh": "2022-04-02T03:30:35.664763Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-02T03:30:35.664763Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "last_refresh": "2022-04-02T03:30:35.664763Z", "running": 0, "size": 1}}] 2022-04-02T03:30:45.865 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph orch ls -f json 2022-04-02T03:30:46.672 INFO:teuthology.orchestra.run.smithi029.stdout: 2022-04-02T03:30:46.673 INFO:teuthology.orchestra.run.smithi029.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:26.000000Z", "last_refresh": "2022-04-02T03:30:46.493591Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:20.000000Z", "last_refresh": "2022-04-02T03:30:46.493591Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:09:56.000000Z", "last_refresh": "2022-04-02T03:30:46.493591Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:15:08.000000Z", "last_refresh": "2022-04-02T03:30:46.493591Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-02T03:30:46.493591Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "last_refresh": "2022-04-02T03:30:46.493591Z", "running": 0, "size": 1}}] 2022-04-02T03:30:56.699 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph orch ls -f json 2022-04-02T03:30:57.379 INFO:teuthology.orchestra.run.smithi029.stdout: 2022-04-02T03:30:57.380 INFO:teuthology.orchestra.run.smithi029.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:26.000000Z", "last_refresh": "2022-04-02T03:30:57.213082Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:20.000000Z", "last_refresh": "2022-04-02T03:30:57.213082Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:09:56.000000Z", "last_refresh": "2022-04-02T03:30:57.213082Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:15:08.000000Z", "last_refresh": "2022-04-02T03:30:57.213082Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-02T03:30:57.213082Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "last_refresh": "2022-04-02T03:30:57.213082Z", "running": 0, "size": 1}}] 2022-04-02T03:31:07.410 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph orch ls -f json 2022-04-02T03:31:08.171 INFO:teuthology.orchestra.run.smithi029.stdout: 2022-04-02T03:31:08.171 INFO:teuthology.orchestra.run.smithi029.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:26.000000Z", "last_refresh": "2022-04-02T03:31:07.911615Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:20.000000Z", "last_refresh": "2022-04-02T03:31:07.911615Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:09:56.000000Z", "last_refresh": "2022-04-02T03:31:07.911615Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:15:08.000000Z", "last_refresh": "2022-04-02T03:31:07.911615Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-02T03:31:07.911615Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "last_refresh": "2022-04-02T03:31:07.911615Z", "running": 0, "size": 1}}] 2022-04-02T03:31:17.441 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:31:17.440343 I | op-mon: parsing mon endpoints: a=10.102.133.199:6789,b=10.107.147.59:6789,c=10.96.208.36:6789 2022-04-02T03:31:18.157 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:31:18.156194 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-04-02T03:31:18.146+0000 7f8bbcaa0380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-04-02T03:31:18.157 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02T03:31:18.146+0000 7f8bbcaa0380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-04-02T03:31:18.157 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02T03:31:18.146+0000 7f8bbcaa0380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-04-02T03:31:18.158 INFO:tasks.rook.operator.smithi029.stdout:couldn't init storage provider. : exit status 5 2022-04-02T03:31:18.196 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph orch ls -f json 2022-04-02T03:31:18.866 INFO:teuthology.orchestra.run.smithi029.stdout: 2022-04-02T03:31:18.866 INFO:teuthology.orchestra.run.smithi029.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:26.000000Z", "last_refresh": "2022-04-02T03:31:18.690512Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:20.000000Z", "last_refresh": "2022-04-02T03:31:18.690512Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:09:56.000000Z", "last_refresh": "2022-04-02T03:31:18.690512Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:15:08.000000Z", "last_refresh": "2022-04-02T03:31:18.690512Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-02T03:31:18.690512Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "last_refresh": "2022-04-02T03:31:18.690512Z", "running": 0, "size": 1}}] 2022-04-02T03:31:28.891 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph orch ls -f json 2022-04-02T03:31:29.625 INFO:teuthology.orchestra.run.smithi029.stdout: 2022-04-02T03:31:29.626 INFO:teuthology.orchestra.run.smithi029.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:26.000000Z", "last_refresh": "2022-04-02T03:31:29.435633Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:20.000000Z", "last_refresh": "2022-04-02T03:31:29.435633Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:09:56.000000Z", "last_refresh": "2022-04-02T03:31:29.435633Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:15:08.000000Z", "last_refresh": "2022-04-02T03:31:29.435633Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-02T03:31:29.435633Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "last_refresh": "2022-04-02T03:31:29.435633Z", "running": 0, "size": 1}}] 2022-04-02T03:31:39.652 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph orch ls -f json 2022-04-02T03:31:40.521 INFO:teuthology.orchestra.run.smithi029.stdout: 2022-04-02T03:31:40.521 INFO:teuthology.orchestra.run.smithi029.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:26.000000Z", "last_refresh": "2022-04-02T03:31:40.316160Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:20.000000Z", "last_refresh": "2022-04-02T03:31:40.316160Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:09:56.000000Z", "last_refresh": "2022-04-02T03:31:40.316160Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:15:08.000000Z", "last_refresh": "2022-04-02T03:31:40.316160Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-02T03:31:40.316160Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "last_refresh": "2022-04-02T03:31:40.316160Z", "running": 0, "size": 1}}] 2022-04-02T03:31:50.544 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph orch ls -f json 2022-04-02T03:31:51.389 INFO:teuthology.orchestra.run.smithi029.stdout: 2022-04-02T03:31:51.390 INFO:teuthology.orchestra.run.smithi029.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:26.000000Z", "last_refresh": "2022-04-02T03:31:51.153613Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:20.000000Z", "last_refresh": "2022-04-02T03:31:51.153613Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:09:56.000000Z", "last_refresh": "2022-04-02T03:31:51.153613Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:15:08.000000Z", "last_refresh": "2022-04-02T03:31:51.153613Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-02T03:31:51.153613Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "last_refresh": "2022-04-02T03:31:51.153613Z", "running": 0, "size": 1}}] 2022-04-02T03:32:01.413 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph orch ls -f json 2022-04-02T03:32:02.108 INFO:teuthology.orchestra.run.smithi029.stdout: 2022-04-02T03:32:02.109 INFO:teuthology.orchestra.run.smithi029.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:26.000000Z", "last_refresh": "2022-04-02T03:32:01.937956Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:20.000000Z", "last_refresh": "2022-04-02T03:32:01.937956Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:09:56.000000Z", "last_refresh": "2022-04-02T03:32:01.937956Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:15:08.000000Z", "last_refresh": "2022-04-02T03:32:01.937956Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-02T03:32:01.937956Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "last_refresh": "2022-04-02T03:32:01.937956Z", "running": 0, "size": 1}}] 2022-04-02T03:32:12.134 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph orch ls -f json 2022-04-02T03:32:12.886 INFO:teuthology.orchestra.run.smithi029.stdout: 2022-04-02T03:32:12.886 INFO:teuthology.orchestra.run.smithi029.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:26.000000Z", "last_refresh": "2022-04-02T03:32:12.701808Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:20.000000Z", "last_refresh": "2022-04-02T03:32:12.701808Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:09:56.000000Z", "last_refresh": "2022-04-02T03:32:12.701808Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:15:08.000000Z", "last_refresh": "2022-04-02T03:32:12.701808Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-02T03:32:12.701808Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "last_refresh": "2022-04-02T03:32:12.701808Z", "running": 0, "size": 1}}] 2022-04-02T03:32:22.910 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph orch ls -f json 2022-04-02T03:32:23.640 INFO:teuthology.orchestra.run.smithi029.stdout: 2022-04-02T03:32:23.640 INFO:teuthology.orchestra.run.smithi029.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:26.000000Z", "last_refresh": "2022-04-02T03:32:23.441760Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:20.000000Z", "last_refresh": "2022-04-02T03:32:23.441760Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:09:56.000000Z", "last_refresh": "2022-04-02T03:32:23.441760Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:15:08.000000Z", "last_refresh": "2022-04-02T03:32:23.441760Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-02T03:32:23.441760Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "last_refresh": "2022-04-02T03:32:23.441760Z", "running": 0, "size": 1}}] 2022-04-02T03:32:33.670 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph orch ls -f json 2022-04-02T03:32:34.389 INFO:teuthology.orchestra.run.smithi029.stdout: 2022-04-02T03:32:34.389 INFO:teuthology.orchestra.run.smithi029.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:26.000000Z", "last_refresh": "2022-04-02T03:32:34.215275Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:20.000000Z", "last_refresh": "2022-04-02T03:32:34.215275Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:09:56.000000Z", "last_refresh": "2022-04-02T03:32:34.215275Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:15:08.000000Z", "last_refresh": "2022-04-02T03:32:34.215275Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-02T03:32:34.215275Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "last_refresh": "2022-04-02T03:32:34.215275Z", "running": 0, "size": 1}}] 2022-04-02T03:32:44.437 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph orch ls -f json 2022-04-02T03:32:45.237 INFO:teuthology.orchestra.run.smithi029.stdout: 2022-04-02T03:32:45.237 INFO:teuthology.orchestra.run.smithi029.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:26.000000Z", "last_refresh": "2022-04-02T03:32:45.067949Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:20.000000Z", "last_refresh": "2022-04-02T03:32:45.067949Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:09:56.000000Z", "last_refresh": "2022-04-02T03:32:45.067949Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:15:08.000000Z", "last_refresh": "2022-04-02T03:32:45.067949Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-02T03:32:45.067949Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "last_refresh": "2022-04-02T03:32:45.067949Z", "running": 0, "size": 1}}] 2022-04-02T03:32:55.261 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph orch ls -f json 2022-04-02T03:32:55.976 INFO:teuthology.orchestra.run.smithi029.stdout: 2022-04-02T03:32:55.976 INFO:teuthology.orchestra.run.smithi029.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:26.000000Z", "last_refresh": "2022-04-02T03:32:55.808084Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:20.000000Z", "last_refresh": "2022-04-02T03:32:55.808084Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:09:56.000000Z", "last_refresh": "2022-04-02T03:32:55.808084Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:15:08.000000Z", "last_refresh": "2022-04-02T03:32:55.808084Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-02T03:32:55.808084Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "last_refresh": "2022-04-02T03:32:55.808084Z", "running": 0, "size": 1}}] 2022-04-02T03:33:06.000 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph orch ls -f json 2022-04-02T03:33:06.751 INFO:teuthology.orchestra.run.smithi029.stdout: 2022-04-02T03:33:06.751 INFO:teuthology.orchestra.run.smithi029.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:26.000000Z", "last_refresh": "2022-04-02T03:33:06.533490Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:20.000000Z", "last_refresh": "2022-04-02T03:33:06.533490Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:09:56.000000Z", "last_refresh": "2022-04-02T03:33:06.533490Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:15:08.000000Z", "last_refresh": "2022-04-02T03:33:06.533490Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-02T03:33:06.533490Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "last_refresh": "2022-04-02T03:33:06.533490Z", "running": 0, "size": 1}}] 2022-04-02T03:33:16.773 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph orch ls -f json 2022-04-02T03:33:17.525 INFO:teuthology.orchestra.run.smithi029.stdout: 2022-04-02T03:33:17.801 INFO:teuthology.orchestra.run.smithi029.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:26.000000Z", "last_refresh": "2022-04-02T03:33:17.294905Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:20.000000Z", "last_refresh": "2022-04-02T03:33:17.294905Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:09:56.000000Z", "last_refresh": "2022-04-02T03:33:17.294905Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:15:08.000000Z", "last_refresh": "2022-04-02T03:33:17.294905Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-02T03:33:17.294905Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "last_refresh": "2022-04-02T03:33:17.294905Z", "running": 0, "size": 1}}] 2022-04-02T03:33:27.803 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph orch ls -f json 2022-04-02T03:33:28.497 INFO:teuthology.orchestra.run.smithi029.stdout: 2022-04-02T03:33:28.498 INFO:teuthology.orchestra.run.smithi029.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:26.000000Z", "last_refresh": "2022-04-02T03:33:28.342999Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:20.000000Z", "last_refresh": "2022-04-02T03:33:28.342999Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:09:56.000000Z", "last_refresh": "2022-04-02T03:33:28.342999Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:15:08.000000Z", "last_refresh": "2022-04-02T03:33:28.342999Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-02T03:33:28.342999Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "last_refresh": "2022-04-02T03:33:28.342999Z", "running": 0, "size": 1}}] 2022-04-02T03:33:38.523 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph orch ls -f json 2022-04-02T03:33:39.234 INFO:teuthology.orchestra.run.smithi029.stdout: 2022-04-02T03:33:39.235 INFO:teuthology.orchestra.run.smithi029.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:26.000000Z", "last_refresh": "2022-04-02T03:33:39.054596Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:20.000000Z", "last_refresh": "2022-04-02T03:33:39.054596Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:09:56.000000Z", "last_refresh": "2022-04-02T03:33:39.054596Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:15:08.000000Z", "last_refresh": "2022-04-02T03:33:39.054596Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-02T03:33:39.054596Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "last_refresh": "2022-04-02T03:33:39.054596Z", "running": 0, "size": 1}}] 2022-04-02T03:33:49.260 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph orch ls -f json 2022-04-02T03:33:49.978 INFO:teuthology.orchestra.run.smithi029.stdout: 2022-04-02T03:33:49.979 INFO:teuthology.orchestra.run.smithi029.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:26.000000Z", "last_refresh": "2022-04-02T03:33:49.778111Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:20.000000Z", "last_refresh": "2022-04-02T03:33:49.778111Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:09:56.000000Z", "last_refresh": "2022-04-02T03:33:49.778111Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:15:08.000000Z", "last_refresh": "2022-04-02T03:33:49.778111Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-02T03:33:49.778111Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "last_refresh": "2022-04-02T03:33:49.778111Z", "running": 0, "size": 1}}] 2022-04-02T03:34:00.004 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph orch ls -f json 2022-04-02T03:34:00.691 INFO:teuthology.orchestra.run.smithi029.stdout: 2022-04-02T03:34:00.691 INFO:teuthology.orchestra.run.smithi029.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:26.000000Z", "last_refresh": "2022-04-02T03:34:00.536813Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:20.000000Z", "last_refresh": "2022-04-02T03:34:00.536813Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:09:56.000000Z", "last_refresh": "2022-04-02T03:34:00.536813Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:15:08.000000Z", "last_refresh": "2022-04-02T03:34:00.536813Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-02T03:34:00.536813Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "last_refresh": "2022-04-02T03:34:00.536813Z", "running": 0, "size": 1}}] 2022-04-02T03:34:10.718 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph orch ls -f json 2022-04-02T03:34:11.504 INFO:teuthology.orchestra.run.smithi029.stdout: 2022-04-02T03:34:11.504 INFO:teuthology.orchestra.run.smithi029.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:26.000000Z", "last_refresh": "2022-04-02T03:34:11.326681Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:20.000000Z", "last_refresh": "2022-04-02T03:34:11.326681Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:09:56.000000Z", "last_refresh": "2022-04-02T03:34:11.326681Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:15:08.000000Z", "last_refresh": "2022-04-02T03:34:11.326681Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-02T03:34:11.326681Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "last_refresh": "2022-04-02T03:34:11.326681Z", "running": 0, "size": 1}}] 2022-04-02T03:34:21.530 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph orch ls -f json 2022-04-02T03:34:22.242 INFO:teuthology.orchestra.run.smithi029.stdout: 2022-04-02T03:34:22.243 INFO:teuthology.orchestra.run.smithi029.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:26.000000Z", "last_refresh": "2022-04-02T03:34:22.056282Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:20.000000Z", "last_refresh": "2022-04-02T03:34:22.056282Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:09:56.000000Z", "last_refresh": "2022-04-02T03:34:22.056282Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:15:08.000000Z", "last_refresh": "2022-04-02T03:34:22.056282Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-02T03:34:22.056282Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "last_refresh": "2022-04-02T03:34:22.056282Z", "running": 0, "size": 1}}] 2022-04-02T03:34:32.271 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph orch ls -f json 2022-04-02T03:34:33.114 INFO:teuthology.orchestra.run.smithi029.stdout: 2022-04-02T03:34:33.115 INFO:teuthology.orchestra.run.smithi029.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:26.000000Z", "last_refresh": "2022-04-02T03:34:32.942223Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:20.000000Z", "last_refresh": "2022-04-02T03:34:32.942223Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:09:56.000000Z", "last_refresh": "2022-04-02T03:34:32.942223Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:15:08.000000Z", "last_refresh": "2022-04-02T03:34:32.942223Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-02T03:34:32.942223Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "last_refresh": "2022-04-02T03:34:32.942223Z", "running": 0, "size": 1}}] 2022-04-02T03:34:43.138 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph orch ls -f json 2022-04-02T03:34:43.886 INFO:teuthology.orchestra.run.smithi029.stdout: 2022-04-02T03:34:43.887 INFO:teuthology.orchestra.run.smithi029.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:26.000000Z", "last_refresh": "2022-04-02T03:34:43.686263Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:20.000000Z", "last_refresh": "2022-04-02T03:34:43.686263Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:09:56.000000Z", "last_refresh": "2022-04-02T03:34:43.686263Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:15:08.000000Z", "last_refresh": "2022-04-02T03:34:43.686263Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-02T03:34:43.686263Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "last_refresh": "2022-04-02T03:34:43.686263Z", "running": 0, "size": 1}}] 2022-04-02T03:34:53.912 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph orch ls -f json 2022-04-02T03:34:54.621 INFO:teuthology.orchestra.run.smithi029.stdout: 2022-04-02T03:34:54.621 INFO:teuthology.orchestra.run.smithi029.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:26.000000Z", "last_refresh": "2022-04-02T03:34:54.425938Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:20.000000Z", "last_refresh": "2022-04-02T03:34:54.425938Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:09:56.000000Z", "last_refresh": "2022-04-02T03:34:54.425938Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:15:08.000000Z", "last_refresh": "2022-04-02T03:34:54.425938Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-02T03:34:54.425938Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "last_refresh": "2022-04-02T03:34:54.425938Z", "running": 0, "size": 1}}] 2022-04-02T03:35:04.648 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph orch ls -f json 2022-04-02T03:35:05.366 INFO:teuthology.orchestra.run.smithi029.stdout: 2022-04-02T03:35:05.367 INFO:teuthology.orchestra.run.smithi029.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:26.000000Z", "last_refresh": "2022-04-02T03:35:05.188251Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:20.000000Z", "last_refresh": "2022-04-02T03:35:05.188251Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:09:56.000000Z", "last_refresh": "2022-04-02T03:35:05.188251Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:15:08.000000Z", "last_refresh": "2022-04-02T03:35:05.188251Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-02T03:35:05.188251Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "last_refresh": "2022-04-02T03:35:05.188251Z", "running": 0, "size": 1}}] 2022-04-02T03:35:15.402 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph orch ls -f json 2022-04-02T03:35:16.129 INFO:teuthology.orchestra.run.smithi029.stdout: 2022-04-02T03:35:16.130 INFO:teuthology.orchestra.run.smithi029.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:26.000000Z", "last_refresh": "2022-04-02T03:35:15.935786Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:20.000000Z", "last_refresh": "2022-04-02T03:35:15.935786Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:09:56.000000Z", "last_refresh": "2022-04-02T03:35:15.935786Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:15:08.000000Z", "last_refresh": "2022-04-02T03:35:15.935786Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-02T03:35:15.935786Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "last_refresh": "2022-04-02T03:35:15.935786Z", "running": 0, "size": 1}}] 2022-04-02T03:35:26.156 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph orch ls -f json 2022-04-02T03:35:26.874 INFO:teuthology.orchestra.run.smithi029.stdout: 2022-04-02T03:35:26.874 INFO:teuthology.orchestra.run.smithi029.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:26.000000Z", "last_refresh": "2022-04-02T03:35:26.670950Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:20.000000Z", "last_refresh": "2022-04-02T03:35:26.670950Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:09:56.000000Z", "last_refresh": "2022-04-02T03:35:26.670950Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:15:08.000000Z", "last_refresh": "2022-04-02T03:35:26.670950Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-02T03:35:26.670950Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "last_refresh": "2022-04-02T03:35:26.670950Z", "running": 0, "size": 1}}] 2022-04-02T03:35:36.902 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph orch ls -f json 2022-04-02T03:35:37.615 INFO:teuthology.orchestra.run.smithi029.stdout: 2022-04-02T03:35:37.616 INFO:teuthology.orchestra.run.smithi029.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:26.000000Z", "last_refresh": "2022-04-02T03:35:37.442822Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:20.000000Z", "last_refresh": "2022-04-02T03:35:37.442822Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:09:56.000000Z", "last_refresh": "2022-04-02T03:35:37.442822Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:15:08.000000Z", "last_refresh": "2022-04-02T03:35:37.442822Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-02T03:35:37.442822Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "last_refresh": "2022-04-02T03:35:37.442822Z", "running": 0, "size": 1}}] 2022-04-02T03:35:47.643 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph orch ls -f json 2022-04-02T03:35:48.394 INFO:teuthology.orchestra.run.smithi029.stdout: 2022-04-02T03:35:48.394 INFO:teuthology.orchestra.run.smithi029.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:26.000000Z", "last_refresh": "2022-04-02T03:35:48.217434Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:20.000000Z", "last_refresh": "2022-04-02T03:35:48.217434Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:09:56.000000Z", "last_refresh": "2022-04-02T03:35:48.217434Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:15:08.000000Z", "last_refresh": "2022-04-02T03:35:48.217434Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-02T03:35:48.217434Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "last_refresh": "2022-04-02T03:35:48.217434Z", "running": 0, "size": 1}}] 2022-04-02T03:35:58.420 DEBUG:teuthology.orchestra.run.smithi029:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-2ddzt -- ceph orch ls -f json 2022-04-02T03:35:59.147 INFO:teuthology.orchestra.run.smithi029.stdout: 2022-04-02T03:35:59.148 INFO:teuthology.orchestra.run.smithi029.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:26.000000Z", "last_refresh": "2022-04-02T03:35:58.979527Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:13:20.000000Z", "last_refresh": "2022-04-02T03:35:58.979527Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:09:56.000000Z", "last_refresh": "2022-04-02T03:35:58.979527Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "a1d2ec729cd28cd0662e273d092773e6999c4d49ccee9bc84a78cbae7eae2cdf", "container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "created": "2022-04-02T03:15:08.000000Z", "last_refresh": "2022-04-02T03:35:58.979527Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-02T03:35:58.979527Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a", "last_refresh": "2022-04-02T03:35:58.979527Z", "running": 0, "size": 1}}] 2022-04-02T03:35:59.172 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_d877b023166308f282187444de8e8e725ffb27d2/teuthology/contextutil.py", line 33, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a/qa/tasks/rook.py", line 669, in task while proceed(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_d877b023166308f282187444de8e8e725ffb27d2/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds 2022-04-02T03:35:59.173 ERROR:tasks.rook:'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a/qa/tasks/rook.py", line 530, in ceph_config_keyring yield File "/home/teuthworker/src/git.ceph.com_git_teuthology_d877b023166308f282187444de8e8e725ffb27d2/teuthology/contextutil.py", line 33, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a/qa/tasks/rook.py", line 669, in task while proceed(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_d877b023166308f282187444de8e8e725ffb27d2/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds 2022-04-02T03:35:59.174 INFO:tasks.rook:Cleaning up config and client.admin keyring 2022-04-02T03:35:59.174 DEBUG:teuthology.orchestra.run.smithi029:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2022-04-02T03:35:59.188 DEBUG:teuthology.orchestra.run.smithi104:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2022-04-02T03:35:59.206 DEBUG:teuthology.orchestra.run.smithi142:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2022-04-02T03:35:59.224 ERROR:tasks.rook:'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a/qa/tasks/rook.py", line 478, in rook_post_config yield File "/home/teuthworker/src/git.ceph.com_git_teuthology_d877b023166308f282187444de8e8e725ffb27d2/teuthology/contextutil.py", line 33, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a/qa/tasks/rook.py", line 669, in task while proceed(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_d877b023166308f282187444de8e8e725ffb27d2/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds 2022-04-02T03:35:59.225 ERROR:tasks.rook:'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a/qa/tasks/rook.py", line 442, in rook_toolbox yield File "/home/teuthworker/src/git.ceph.com_git_teuthology_d877b023166308f282187444de8e8e725ffb27d2/teuthology/contextutil.py", line 33, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a/qa/tasks/rook.py", line 669, in task while proceed(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_d877b023166308f282187444de8e8e725ffb27d2/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds 2022-04-02T03:35:59.236 DEBUG:teuthology.orchestra.remote:smithi029:rook/deploy/examples/operator.yaml is 20KB 2022-04-02T03:35:59.248 DEBUG:teuthology.orchestra.run.smithi029:> kubectl delete -f rook/deploy/examples/toolbox.yaml 2022-04-02T03:35:59.316 INFO:teuthology.orchestra.run.smithi029.stdout:deployment.apps "rook-ceph-tools" deleted 2022-04-02T03:35:59.325 ERROR:tasks.rook:'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a/qa/tasks/rook.py", line 379, in rook_cluster yield File "/home/teuthworker/src/git.ceph.com_git_teuthology_d877b023166308f282187444de8e8e725ffb27d2/teuthology/contextutil.py", line 33, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_76cd7b384b2b46da30d83ba18f236e3ddc3bdb8a/qa/tasks/rook.py", line 669, in task while proceed(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_d877b023166308f282187444de8e8e725ffb27d2/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds 2022-04-02T03:35:59.326 DEBUG:teuthology.orchestra.run.smithi029:> kubectl delete -f cluster.yaml 2022-04-02T03:35:59.401 INFO:teuthology.orchestra.run.smithi029.stdout:cephcluster.ceph.rook.io "rook-ceph" deleted 2022-04-02T03:35:59.408 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:35:59.405511 I | ceph-cluster-controller: CR "rook-ceph" is going be deleted, cancelling any ongoing orchestration 2022-04-02T03:35:59.409 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:35:59.405610 I | operator: reloading operator's CRDs manager, cancelling all orchestrations! 2022-04-02T03:35:59.409 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:35:59.405672 I | ceph-cluster-controller: stopping monitoring of ceph status 2022-04-02T03:35:59.410 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:35:59.405704 I | op-mon: stopping monitoring of mons in namespace "rook-ceph" 2022-04-02T03:35:59.410 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:35:59.405719 I | op-osd: stopping monitoring of OSDs in namespace "rook-ceph" 2022-04-02T03:35:59.410 INFO:tasks.rook.operator.smithi029.stdout:I0402 03:35:59.405955 1 manager.go:148] objectbucket.io/provisioner-manager "msg"="stopping provisioner" "name"="rook-ceph.ceph.rook.io/bucket" "reason"={} 2022-04-02T03:35:59.410 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:35:59.406053 I | operator: successfully started the controller-runtime manager 2022-04-02T03:35:59.455 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:35:59.455192 I | op-k8sutil: ROOK_CURRENT_NAMESPACE_ONLY="false" (env var) 2022-04-02T03:35:59.456 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:35:59.455221 I | operator: watching all namespaces for Ceph CRs 2022-04-02T03:35:59.456 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:35:59.455255 I | operator: setting up schemes 2022-04-02T03:35:59.459 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:35:59.459179 I | operator: setting up the controller-runtime manager 2022-04-02T03:35:59.663 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:35:59.663446 I | operator: looking for admission webhook secret "rook-ceph-admission-controller" 2022-04-02T03:35:59.669 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:35:59.666292 I | operator: admission webhook secret "rook-ceph-admission-controller" not found. proceeding without the admission controller 2022-04-02T03:35:59.669 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:35:59.666338 I | ceph-cluster-controller: successfully started 2022-04-02T03:35:59.669 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:35:59.666406 I | ceph-cluster-controller: enabling hotplug orchestration 2022-04-02T03:35:59.670 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:35:59.666436 I | ceph-crashcollector-controller: successfully started 2022-04-02T03:35:59.671 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:35:59.666466 I | ceph-block-pool-controller: successfully started 2022-04-02T03:35:59.671 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:35:59.666492 I | ceph-object-store-user-controller: successfully started 2022-04-02T03:35:59.671 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:35:59.666521 I | ceph-object-realm-controller: successfully started 2022-04-02T03:35:59.671 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:35:59.666542 I | ceph-object-zonegroup-controller: successfully started 2022-04-02T03:35:59.672 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:35:59.666556 I | ceph-object-zone-controller: successfully started 2022-04-02T03:35:59.672 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:35:59.666727 I | ceph-object-controller: successfully started 2022-04-02T03:35:59.672 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:35:59.666783 I | ceph-file-controller: successfully started 2022-04-02T03:35:59.672 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:35:59.666824 I | ceph-nfs-controller: successfully started 2022-04-02T03:35:59.673 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:35:59.666866 I | ceph-rbd-mirror-controller: successfully started 2022-04-02T03:35:59.673 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:35:59.666897 I | ceph-client-controller: successfully started 2022-04-02T03:35:59.673 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:35:59.666923 I | ceph-filesystem-mirror-controller: successfully started 2022-04-02T03:35:59.673 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:35:59.666959 I | operator: rook-ceph-operator-config-controller successfully started 2022-04-02T03:35:59.674 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:35:59.666982 I | ceph-csi: rook-ceph-operator-csi-controller successfully started 2022-04-02T03:35:59.674 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:35:59.667004 I | op-bucket-prov: rook-ceph-operator-bucket-controller successfully started 2022-04-02T03:35:59.674 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:35:59.667036 I | ceph-bucket-topic: successfully started 2022-04-02T03:35:59.674 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:35:59.667057 I | ceph-bucket-notification: successfully started 2022-04-02T03:35:59.674 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:35:59.667072 I | ceph-bucket-notification: successfully started 2022-04-02T03:35:59.675 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:35:59.667085 I | ceph-fs-subvolumegroup-controller: successfully started 2022-04-02T03:35:59.675 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:35:59.668176 I | operator: starting the controller-runtime manager 2022-04-02T03:35:59.773 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:35:59.773078 I | op-k8sutil: ROOK_CEPH_COMMANDS_TIMEOUT_SECONDS="15" (configmap) 2022-04-02T03:35:59.777 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:35:59.773157 I | op-k8sutil: ROOK_LOG_LEVEL="INFO" (configmap) 2022-04-02T03:35:59.777 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:35:59.773169 I | op-k8sutil: ROOK_ENABLE_DISCOVERY_DAEMON="false" (configmap) 2022-04-02T03:35:59.778 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:35:59.776743 I | operator: rook-ceph-operator-config-controller done reconciling 2022-04-02T03:35:59.780 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:35:59.779174 I | op-mon: parsing mon endpoints: a=10.102.133.199:6789,b=10.107.147.59:6789,c=10.96.208.36:6789 2022-04-02T03:35:59.853 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:35:59.853590 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:35:59.904 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:35:59.903984 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:36:00.427 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:36:00.426487 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-04-02T03:36:00.413+0000 7f9e95e5c380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-04-02T03:36:00.427 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02T03:36:00.413+0000 7f9e95e5c380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-04-02T03:36:00.427 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02T03:36:00.413+0000 7f9e95e5c380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-04-02T03:36:00.428 INFO:tasks.rook.operator.smithi029.stdout:couldn't init storage provider. : exit status 5 2022-04-02T03:36:00.442 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:36:00.442851 I | op-mon: parsing mon endpoints: a=10.102.133.199:6789,b=10.107.147.59:6789,c=10.96.208.36:6789 2022-04-02T03:36:01.146 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:36:01.145837 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-04-02T03:36:01.133+0000 7f16b3d1b380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-04-02T03:36:01.147 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02T03:36:01.133+0000 7f16b3d1b380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-04-02T03:36:01.147 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02T03:36:01.133+0000 7f16b3d1b380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-04-02T03:36:01.147 INFO:tasks.rook.operator.smithi029.stdout:couldn't init storage provider. : exit status 5 2022-04-02T03:36:01.162 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:36:01.161869 I | op-mon: parsing mon endpoints: a=10.102.133.199:6789,b=10.107.147.59:6789,c=10.96.208.36:6789 2022-04-02T03:36:01.813 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:36:01.812519 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-04-02T03:36:01.801+0000 7f3caab5f380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-04-02T03:36:01.813 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02T03:36:01.801+0000 7f3caab5f380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-04-02T03:36:01.813 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02T03:36:01.801+0000 7f3caab5f380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-04-02T03:36:01.814 INFO:tasks.rook.operator.smithi029.stdout:couldn't init storage provider. : exit status 5 2022-04-02T03:36:01.879 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:36:01.879633 I | op-mon: parsing mon endpoints: a=10.102.133.199:6789,b=10.107.147.59:6789,c=10.96.208.36:6789 2022-04-02T03:36:02.624 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:36:02.624086 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-04-02T03:36:02.613+0000 7fe47083a380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-04-02T03:36:02.625 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02T03:36:02.613+0000 7fe47083a380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-04-02T03:36:02.625 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02T03:36:02.613+0000 7fe47083a380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-04-02T03:36:02.625 INFO:tasks.rook.operator.smithi029.stdout:couldn't init storage provider. : exit status 5 2022-04-02T03:36:02.690 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:36:02.690515 I | op-mon: parsing mon endpoints: a=10.102.133.199:6789,b=10.107.147.59:6789,c=10.96.208.36:6789 2022-04-02T03:36:03.384 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:36:03.383988 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-04-02T03:36:03.369+0000 7f34e9086380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-04-02T03:36:03.385 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02T03:36:03.369+0000 7f34e9086380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-04-02T03:36:03.385 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02T03:36:03.369+0000 7f34e9086380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-04-02T03:36:03.385 INFO:tasks.rook.operator.smithi029.stdout:couldn't init storage provider. : exit status 5 2022-04-02T03:36:03.471 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:36:03.470764 I | op-mon: parsing mon endpoints: a=10.102.133.199:6789,b=10.107.147.59:6789,c=10.96.208.36:6789 2022-04-02T03:36:04.106 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:36:04.105454 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-04-02T03:36:04.093+0000 7f3d7a2c4380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-04-02T03:36:04.106 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02T03:36:04.093+0000 7f3d7a2c4380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-04-02T03:36:04.107 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02T03:36:04.093+0000 7f3d7a2c4380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-04-02T03:36:04.107 INFO:tasks.rook.operator.smithi029.stdout:couldn't init storage provider. : exit status 5 2022-04-02T03:36:04.289 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:36:04.289216 I | op-mon: parsing mon endpoints: a=10.102.133.199:6789,b=10.107.147.59:6789,c=10.96.208.36:6789 2022-04-02T03:36:04.980 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:36:04.980171 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-04-02T03:36:04.965+0000 7f7ebafd4380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-04-02T03:36:04.980 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02T03:36:04.965+0000 7f7ebafd4380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-04-02T03:36:04.981 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02T03:36:04.965+0000 7f7ebafd4380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-04-02T03:36:04.981 INFO:tasks.rook.operator.smithi029.stdout:couldn't init storage provider. : exit status 5 2022-04-02T03:36:05.308 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:36:05.307750 I | op-mon: parsing mon endpoints: a=10.102.133.199:6789,b=10.107.147.59:6789,c=10.96.208.36:6789 2022-04-02T03:36:05.979 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:36:05.978983 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-04-02T03:36:05.965+0000 7f90eec57380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-04-02T03:36:05.979 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02T03:36:05.965+0000 7f90eec57380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-04-02T03:36:05.980 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02T03:36:05.965+0000 7f90eec57380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-04-02T03:36:05.980 INFO:tasks.rook.operator.smithi029.stdout:couldn't init storage provider. : exit status 5 2022-04-02T03:36:06.639 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:36:06.639057 I | op-mon: parsing mon endpoints: a=10.102.133.199:6789,b=10.107.147.59:6789,c=10.96.208.36:6789 2022-04-02T03:36:07.308 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:36:07.307854 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-04-02T03:36:07.293+0000 7f952ccaa380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-04-02T03:36:07.308 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02T03:36:07.293+0000 7f952ccaa380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-04-02T03:36:07.309 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02T03:36:07.293+0000 7f952ccaa380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-04-02T03:36:07.309 INFO:tasks.rook.operator.smithi029.stdout:couldn't init storage provider. : exit status 5 2022-04-02T03:36:08.595 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:36:08.594785 I | op-mon: parsing mon endpoints: a=10.102.133.199:6789,b=10.107.147.59:6789,c=10.96.208.36:6789 2022-04-02T03:36:09.292 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:36:09.291572 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-04-02T03:36:09.277+0000 7f92a7e8f380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-04-02T03:36:09.292 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02T03:36:09.277+0000 7f92a7e8f380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-04-02T03:36:09.292 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02T03:36:09.277+0000 7f92a7e8f380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-04-02T03:36:09.293 INFO:tasks.rook.operator.smithi029.stdout:couldn't init storage provider. : exit status 5 2022-04-02T03:36:09.968 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:36:09.967585 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:36:09.982 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:36:09.982373 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:36:11.859 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:36:11.859481 I | op-mon: parsing mon endpoints: a=10.102.133.199:6789,b=10.107.147.59:6789,c=10.96.208.36:6789 2022-04-02T03:36:12.559 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:36:12.558560 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-04-02T03:36:12.545+0000 7f75e0b0a380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-04-02T03:36:12.559 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02T03:36:12.545+0000 7f75e0b0a380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-04-02T03:36:12.559 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02T03:36:12.545+0000 7f75e0b0a380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-04-02T03:36:12.560 INFO:tasks.rook.operator.smithi029.stdout:couldn't init storage provider. : exit status 5 2022-04-02T03:36:17.689 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:36:17.688615 I | op-mon: parsing mon endpoints: a=10.102.133.199:6789,b=10.107.147.59:6789,c=10.96.208.36:6789 2022-04-02T03:36:18.395 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:36:18.394641 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-04-02T03:36:18.385+0000 7f76cd5df380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-04-02T03:36:18.395 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02T03:36:18.385+0000 7f76cd5df380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-04-02T03:36:18.396 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02T03:36:18.385+0000 7f76cd5df380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-04-02T03:36:18.396 INFO:tasks.rook.operator.smithi029.stdout:couldn't init storage provider. : exit status 5 2022-04-02T03:36:20.104 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:36:20.104366 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:36:20.116 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:36:20.116456 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:36:28.641 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:36:28.640900 I | op-mon: parsing mon endpoints: a=10.102.133.199:6789,b=10.107.147.59:6789,c=10.96.208.36:6789 2022-04-02T03:36:29.289 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:36:29.288798 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-04-02T03:36:29.280+0000 7fa8e97ec380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-04-02T03:36:29.289 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02T03:36:29.280+0000 7fa8e97ec380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-04-02T03:36:29.289 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02T03:36:29.280+0000 7fa8e97ec380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-04-02T03:36:29.290 INFO:tasks.rook.operator.smithi029.stdout:couldn't init storage provider. : exit status 5 2022-04-02T03:36:30.224 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:36:30.224151 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:36:30.237 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:36:30.237037 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:36:40.311 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:36:40.311494 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:36:40.361 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:36:40.361328 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:36:49.776 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:36:49.776428 I | op-mon: parsing mon endpoints: a=10.102.133.199:6789,b=10.107.147.59:6789,c=10.96.208.36:6789 2022-04-02T03:36:50.427 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:36:50.427617 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-04-02T03:36:50.416+0000 7f7b5924b380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-04-02T03:36:50.428 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02T03:36:50.416+0000 7f7b5924b380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-04-02T03:36:50.428 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02T03:36:50.416+0000 7f7b5924b380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-04-02T03:36:50.429 INFO:tasks.rook.operator.smithi029.stdout:couldn't init storage provider. : exit status 5 2022-04-02T03:36:50.432 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:36:50.430275 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:36:50.526 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:36:50.525832 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:37:00.610 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:37:00.609769 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:37:00.622 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:37:00.622839 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:37:10.757 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:37:10.756654 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:37:10.769 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:37:10.769028 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:37:20.886 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:37:20.885727 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:37:20.899 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:37:20.898793 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:37:31.011 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:37:31.011344 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:37:31.025 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:37:31.025048 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:37:31.395 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:37:31.395088 I | op-mon: parsing mon endpoints: a=10.102.133.199:6789,b=10.107.147.59:6789,c=10.96.208.36:6789 2022-04-02T03:37:32.054 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:37:32.053863 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-04-02T03:37:32.043+0000 7fe5c8a77380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-04-02T03:37:32.054 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02T03:37:32.043+0000 7fe5c8a77380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-04-02T03:37:32.055 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02T03:37:32.043+0000 7fe5c8a77380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-04-02T03:37:32.055 INFO:tasks.rook.operator.smithi029.stdout:couldn't init storage provider. : exit status 5 2022-04-02T03:37:41.088 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:37:41.088310 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:37:41.101 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:37:41.100793 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:37:51.216 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:37:51.215437 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:37:51.228 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:37:51.228516 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:38:01.358 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:38:01.357673 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:38:01.369 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:38:01.369565 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:38:11.494 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:38:11.494034 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:38:11.507 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:38:11.506597 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:38:21.586 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:38:21.585603 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:38:21.597 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:38:21.596843 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:38:31.739 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:38:31.738070 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:38:31.751 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:38:31.750669 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:38:41.829 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:38:41.828908 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:38:41.843 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:38:41.842895 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:38:51.916 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:38:51.915942 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:38:51.930 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:38:51.930233 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:38:53.980 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:38:53.980161 I | op-mon: parsing mon endpoints: a=10.102.133.199:6789,b=10.107.147.59:6789,c=10.96.208.36:6789 2022-04-02T03:38:54.815 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:38:54.814837 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-04-02T03:38:54.801+0000 7f0e2f325380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-04-02T03:38:54.816 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02T03:38:54.801+0000 7f0e2f325380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-04-02T03:38:54.816 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02T03:38:54.801+0000 7f0e2f325380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-04-02T03:38:54.816 INFO:tasks.rook.operator.smithi029.stdout:couldn't init storage provider. : exit status 5 2022-04-02T03:39:02.033 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:39:02.032265 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:39:02.046 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:39:02.046208 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:39:12.115 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:39:12.114671 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:39:12.128 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:39:12.128197 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:39:22.249 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:39:22.248857 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:39:22.287 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:39:22.286692 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:39:32.392 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:39:32.391548 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:39:32.406 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:39:32.406215 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:39:42.475 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:39:42.474866 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:39:42.488 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:39:42.488567 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:39:52.582 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:39:52.581523 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:39:52.595 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:39:52.595489 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:40:02.669 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:40:02.668403 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:40:02.683 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:40:02.683079 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:40:12.774 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:40:12.773726 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:40:12.869 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:40:12.868948 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:40:23.067 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:40:23.067225 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:40:23.180 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:40:23.179825 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:40:33.286 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:40:33.285179 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:40:33.297 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:40:33.297536 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:40:43.427 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:40:43.427053 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:40:43.439 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:40:43.439267 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:40:53.521 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:40:53.521025 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:40:53.539 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:40:53.538757 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:41:03.657 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:41:03.656828 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:41:03.670 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:41:03.670147 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:41:13.785 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:41:13.784680 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:41:13.801 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:41:13.800895 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:41:23.873 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:41:23.873307 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:41:23.915 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:41:23.911148 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:41:34.011 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:41:34.010938 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:41:34.024 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:41:34.023921 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:41:38.675 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:41:38.674450 I | op-mon: parsing mon endpoints: a=10.102.133.199:6789,b=10.107.147.59:6789,c=10.96.208.36:6789 2022-04-02T03:41:39.376 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:41:39.376096 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-04-02T03:41:39.366+0000 7f172a55e380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-04-02T03:41:39.377 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02T03:41:39.366+0000 7f172a55e380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-04-02T03:41:39.377 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02T03:41:39.366+0000 7f172a55e380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-04-02T03:41:39.377 INFO:tasks.rook.operator.smithi029.stdout:couldn't init storage provider. : exit status 5 2022-04-02T03:41:44.140 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:41:44.139779 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:41:44.153 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:41:44.152964 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:41:54.227 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:41:54.227391 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:41:54.280 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:41:54.279627 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:42:04.430 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:42:04.429830 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:42:04.444 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:42:04.443850 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:42:14.585 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:42:14.585263 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:42:14.595 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:42:14.595345 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:42:24.834 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:42:24.833551 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:42:24.847 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:42:24.847403 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:42:35.036 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:42:35.035434 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:42:35.051 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:42:35.051017 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:42:45.114 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:42:45.113965 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:42:45.218 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:42:45.218242 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:42:55.534 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:42:55.533870 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:42:55.549 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:42:55.549559 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:43:05.775 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:43:05.774784 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:43:05.792 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:43:05.791952 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:43:15.956 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:43:15.955993 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:43:15.970 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:43:15.969851 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:43:26.070 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:43:26.070053 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:43:26.084 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:43:26.084074 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:43:36.192 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:43:36.191487 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:43:36.207 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:43:36.207033 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:43:46.313 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:43:46.313315 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:43:46.328 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:43:46.328477 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:43:56.402 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:43:56.401495 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:43:56.416 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:43:56.416370 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:44:06.516 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:44:06.516037 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:44:06.530 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:44:06.529985 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:44:16.637 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:44:16.637377 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:44:16.653 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:44:16.653009 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:44:26.763 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:44:26.762971 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:44:26.777 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:44:26.777666 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:44:36.886 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:44:36.885667 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:44:36.901 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:44:36.900769 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:44:46.981 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:44:46.980852 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:44:46.995 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:44:46.995106 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:44:57.129 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:44:57.129005 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:44:57.144 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:44:57.144818 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:45:07.245 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:45:07.245535 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:45:07.261 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:45:07.261356 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:45:17.369 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:45:17.368845 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:45:17.382 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:45:17.382439 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:45:27.493 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:45:27.492485 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:45:27.507 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:45:27.507205 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:45:37.582 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:45:37.582106 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:45:37.597 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:45:37.597398 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:45:47.717 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:45:47.716580 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:45:47.730 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:45:47.729898 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:45:57.844 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:45:57.843957 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:45:57.856 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:45:57.856026 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:46:07.928 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:46:07.928076 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:46:07.943 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:46:07.943121 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:46:18.072 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:46:18.072056 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:46:18.084 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:46:18.084186 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:46:28.228 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:46:28.228102 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:46:28.239 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:46:28.239955 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:46:38.345 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:46:38.345282 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:46:38.360 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:46:38.360041 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:46:48.486 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:46:48.486111 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:46:48.499 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:46:48.499653 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:46:58.604 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:46:58.603548 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:46:58.616 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:46:58.616829 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:47:07.070 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:47:07.070515 I | op-mon: parsing mon endpoints: a=10.102.133.199:6789,b=10.107.147.59:6789,c=10.96.208.36:6789 2022-04-02T03:47:07.717 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:47:07.716477 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-04-02T03:47:07.708+0000 7f0a05bdd380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-04-02T03:47:07.717 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02T03:47:07.708+0000 7f0a05bdd380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-04-02T03:47:07.717 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02T03:47:07.708+0000 7f0a05bdd380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-04-02T03:47:07.718 INFO:tasks.rook.operator.smithi029.stdout:couldn't init storage provider. : exit status 5 2022-04-02T03:47:08.704 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:47:08.703649 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:47:08.723 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:47:08.719365 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:47:18.831 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:47:18.830905 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:47:18.847 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:47:18.847367 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:47:28.939 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:47:28.939622 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:47:28.952 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:47:28.952832 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:47:39.021 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:47:39.021511 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:47:39.033 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:47:39.033629 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:47:49.148 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:47:49.148684 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:47:49.161 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:47:49.161064 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:47:59.295 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:47:59.295409 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:47:59.309 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:47:59.309983 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:48:09.405 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:48:09.404846 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:48:09.419 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:48:09.419531 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:48:19.496 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:48:19.496138 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:48:19.511 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:48:19.510941 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:48:29.638 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:48:29.638264 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:48:29.652 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:48:29.652493 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:48:39.739 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:48:39.739386 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:48:39.780 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:48:39.780364 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:48:49.901 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:48:49.901572 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:48:49.915 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:48:49.914799 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:49:00.023 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:49:00.023015 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:49:00.036 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:49:00.037005 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:49:10.110 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:49:10.110390 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:49:10.124 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:49:10.124791 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:49:20.237 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:49:20.237476 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:49:20.255 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:49:20.255772 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:49:30.354 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:49:30.354587 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:49:30.367 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:49:30.367373 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:49:40.444 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:49:40.444452 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:49:40.458 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:49:40.459108 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:49:50.520 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:49:50.520479 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:49:50.533 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:49:50.533827 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:50:00.646 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:50:00.646334 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:50:00.659 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:50:00.659241 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:50:10.789 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:50:10.788916 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:50:10.804 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:50:10.804364 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:50:20.908 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:50:20.908820 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:50:20.937 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:50:20.927336 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:50:31.026 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:50:31.026173 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:50:31.041 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:50:31.042180 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:50:41.116 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:50:41.117008 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:50:41.129 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:50:41.130155 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:50:51.226 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:50:51.226119 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:50:51.239 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:50:51.240292 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:51:01.344 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:51:01.344638 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:51:01.359 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:51:01.360038 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:51:11.425 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:51:11.426145 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:51:11.439 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:51:11.439262 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:51:21.509 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:51:21.510044 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:51:21.583 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:51:21.583695 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:51:31.747 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:51:31.747795 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:51:31.759 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:51:31.760331 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:51:41.875 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:51:41.875413 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:51:41.889 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:51:41.890253 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:51:52.003 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:51:52.003441 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:51:52.017 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:51:52.017502 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:52:02.124 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:52:02.124753 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:52:02.144 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:52:02.145116 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:52:12.251 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:52:12.251736 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:52:12.265 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:52:12.266204 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:52:22.332 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:52:22.328808 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:52:22.343 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:52:22.343770 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:52:32.460 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:52:32.460861 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:52:32.476 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:52:32.474634 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:52:42.562 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:52:42.562793 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:52:42.571 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:52:42.572581 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:52:52.644 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:52:52.644273 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:52:52.654 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:52:52.654699 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:53:02.777 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:53:02.777063 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:53:02.792 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:53:02.793230 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:53:12.908 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:53:12.909355 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:53:12.921 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:53:12.922019 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:53:22.990 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:53:22.991134 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:53:23.006 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:53:23.007101 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:53:33.107 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:53:33.107646 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:53:33.120 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:53:33.121207 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:53:43.193 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:53:43.192908 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:53:43.312 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:53:43.208281 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:53:53.308 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:53:53.308892 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:53:53.325 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:53:53.326332 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:54:06.175 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:54:06.175601 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:54:06.186 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:54:06.187526 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:54:16.249 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:54:16.249544 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:54:16.308 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:54:16.308949 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:54:26.427 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:54:26.428268 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:54:26.441 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:54:26.441551 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:54:36.541 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:54:36.542056 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:54:36.555 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:54:36.555962 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:54:46.662 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:54:46.662739 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:54:46.674 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:54:46.675224 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:54:56.771 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:54:56.771596 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:54:56.784 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:54:56.784952 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:55:06.859 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:55:06.859567 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:55:06.874 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:55:06.874636 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:55:16.953 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:55:16.953735 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:55:16.965 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:55:16.965869 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:55:27.025 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:55:27.025376 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:55:27.076 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:55:27.076885 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:55:37.171 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:55:37.171458 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:55:37.185 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:55:37.185998 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:55:47.294 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:55:47.294882 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:55:47.305 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:55:47.305859 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:55:57.424 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:55:57.424809 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:55:57.436 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:55:57.436876 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:56:07.531 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:56:07.531609 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:56:07.543 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:56:07.543907 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:56:17.641 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:56:17.641784 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:56:17.657 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:56:17.657898 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:56:27.733 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:56:27.733659 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:56:27.804 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:56:27.804571 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:56:37.959 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:56:37.959728 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:56:37.993 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:56:37.993949 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:56:48.070 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:56:48.070117 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:56:48.131 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:56:48.131622 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:56:58.230 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:56:58.230416 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:56:58.242 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:56:58.243245 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:57:08.318 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:57:08.318528 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:57:08.331 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:57:08.331634 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:57:18.434 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:57:18.434019 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:57:18.448 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:57:18.448696 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:57:28.563 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:57:28.563118 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:57:28.575 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:57:28.575926 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:57:38.686 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:57:38.686148 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:57:38.698 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:57:38.699332 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:57:48.857 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:57:48.857919 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:57:48.871 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:57:48.872302 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:57:58.995 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:57:58.995476 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:57:59.004 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:57:59.005425 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:58:03.085 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:58:03.086003 I | op-mon: parsing mon endpoints: a=10.102.133.199:6789,b=10.107.147.59:6789,c=10.96.208.36:6789 2022-04-02T03:58:03.793 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:58:03.793975 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-04-02T03:58:03.785+0000 7f791f834380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-04-02T03:58:03.794 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02T03:58:03.785+0000 7f791f834380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-04-02T03:58:03.794 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02T03:58:03.785+0000 7f791f834380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-04-02T03:58:03.794 INFO:tasks.rook.operator.smithi029.stdout:couldn't init storage provider. : exit status 5 2022-04-02T03:58:09.076 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:58:09.076814 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:58:09.087 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:58:09.087846 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:58:19.153 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:58:19.154053 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:58:19.173 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:58:19.173725 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:58:29.308 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:58:29.308783 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:58:29.319 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:58:29.319895 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:58:39.435 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:58:39.436001 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:58:39.448 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:58:39.448709 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:58:49.570 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:58:49.570573 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:58:49.584 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:58:49.584400 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:58:59.720 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:58:59.720036 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:58:59.734 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:58:59.734608 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:59:09.834 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:59:09.834644 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:59:09.847 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:59:09.847594 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:59:19.915 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:59:19.915183 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:59:19.931 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:59:19.931956 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:59:30.006 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:59:30.006337 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:59:30.078 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:59:30.078785 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:59:40.184 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:59:40.184513 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:59:40.199 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:59:40.199823 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:59:50.310 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:59:50.310941 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T03:59:50.324 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 03:59:50.324721 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:00:00.413 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:00:00.414199 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:00:00.424 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:00:00.425298 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:00:10.501 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:00:10.501457 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:00:10.517 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:00:10.517279 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:00:20.694 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:00:20.695076 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:00:20.707 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:00:20.707693 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:00:30.780 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:00:30.780389 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:00:30.885 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:00:30.885864 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:00:40.959 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:00:40.958607 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:00:40.972 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:00:40.972601 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:00:51.055 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:00:51.055395 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:00:51.067 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:00:51.067565 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:01:01.183 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:01:01.183939 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:01:01.197 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:01:01.198264 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:01:11.276 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:01:11.276715 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:01:11.358 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:01:11.358728 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:01:21.431 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:01:21.431973 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:01:21.503 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:01:21.503216 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:01:31.612 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:01:31.611726 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:01:31.639 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:01:31.628133 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:01:41.695 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:01:41.695795 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:01:41.710 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:01:41.710388 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:01:51.808 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:01:51.808836 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:01:51.824 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:01:51.824784 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:02:01.895 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:02:01.895200 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:02:01.907 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:02:01.908164 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:02:12.052 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:02:12.052075 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:02:12.068 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:02:12.068482 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:02:22.158 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:02:22.158764 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:02:22.172 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:02:22.172799 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:02:32.282 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:02:32.282590 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:02:32.296 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:02:32.296612 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:02:42.430 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:02:42.430750 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:02:42.444 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:02:42.444466 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:02:52.521 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:02:52.521086 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:02:52.533 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:02:52.533903 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:03:02.607 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:03:02.603704 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:03:02.619 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:03:02.618988 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:03:12.708 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:03:12.708184 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:03:12.720 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:03:12.720243 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:03:22.808 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:03:22.808743 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:03:22.821 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:03:22.821556 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:03:32.889 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:03:32.889608 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:03:32.948 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:03:32.948511 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:03:43.078 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:03:43.078565 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:03:43.091 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:03:43.090969 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:03:53.165 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:03:53.164795 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:03:53.178 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:03:53.178139 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:04:03.331 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:04:03.330945 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:04:03.343 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:04:03.343259 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:04:13.479 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:04:13.478586 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:04:13.489 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:04:13.489508 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:04:23.695 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:04:23.694679 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:04:23.706 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:04:23.706761 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:04:33.780 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:04:33.780052 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:04:33.814 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:04:33.814213 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:04:43.885 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:04:43.884905 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:04:43.899 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:04:43.899772 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:04:54.023 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:04:54.022607 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:04:54.033 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:04:54.033281 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:05:04.192 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:05:04.192061 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:05:04.206 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:05:04.206120 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:05:14.357 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:05:14.356794 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:05:14.373 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:05:14.372926 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:05:24.435 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:05:24.435543 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:05:24.446 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:05:24.446217 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:05:34.554 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:05:34.553606 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:05:34.567 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:05:34.567689 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:05:44.676 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:05:44.675794 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:05:44.689 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:05:44.689164 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:05:54.763 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:05:54.763387 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:05:54.775 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:05:54.775238 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:06:04.845 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:06:04.845200 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:06:04.862 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:06:04.861973 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:06:14.941 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:06:14.940587 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:06:14.954 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:06:14.953905 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:06:25.024 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:06:25.023838 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:06:25.038 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:06:25.037450 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:06:35.114 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:06:35.114182 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:06:35.127 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:06:35.127677 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:06:45.235 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:06:45.235010 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:06:45.249 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:06:45.249663 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:06:55.332 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:06:55.331628 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:06:55.344 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:06:55.343947 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:07:05.553 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:07:05.552665 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:07:05.565 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:07:05.565678 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:07:15.683 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:07:15.682282 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:07:15.699 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:07:15.698687 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:07:25.907 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:07:25.907269 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:07:25.918 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:07:25.917954 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:07:35.991 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:07:35.991161 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:07:36.058 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:07:36.057910 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:07:46.216 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:07:46.215699 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:07:46.231 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:07:46.231182 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:07:56.303 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:07:56.303401 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:07:56.317 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:07:56.316893 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:08:06.423 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:08:06.422530 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:08:06.435 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:08:06.435240 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:08:16.542 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:08:16.541911 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:08:16.558 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:08:16.556127 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:08:26.624 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:08:26.623426 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:08:26.639 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:08:26.638713 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:08:36.744 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:08:36.743318 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:08:36.774 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:08:36.773787 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:08:46.844 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:08:46.844153 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:08:46.860 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:08:46.860101 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:08:56.963 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:08:56.963071 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:08:56.978 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:08:56.978007 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:09:07.078 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:09:07.077465 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:09:07.091 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:09:07.091000 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:09:17.224 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:09:17.223401 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:09:17.236 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:09:17.236572 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:09:27.348 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:09:27.347541 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:09:27.363 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:09:27.363302 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:09:37.483 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:09:37.482691 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:09:37.497 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:09:37.497203 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:09:47.604 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:09:47.598271 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:09:47.612 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:09:47.611847 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:09:57.751 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:09:57.750660 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:09:57.764 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:09:57.764311 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:10:07.872 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:10:07.872412 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:10:07.885 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:10:07.884916 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:10:18.013 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:10:18.013074 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:10:18.041 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:10:18.040134 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:10:28.145 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:10:28.145298 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:10:28.158 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:10:28.158140 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:10:38.275 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:10:38.272858 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:10:38.286 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:10:38.285900 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:10:48.366 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:10:48.365746 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:10:48.379 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:10:48.378970 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:10:58.482 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:10:58.481904 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:10:58.498 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:10:58.498134 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:11:08.577 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:11:08.576723 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:11:08.591 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:11:08.591219 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:11:18.707 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:11:18.706470 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:11:18.724 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:11:18.723955 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:11:28.828 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:11:28.827602 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:11:28.843 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:11:28.842635 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:11:38.912 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:11:38.911489 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:11:38.926 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:11:38.925962 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:11:49.003 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:11:49.002298 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:11:49.022 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:11:49.021822 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:11:59.124 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:11:59.123530 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:11:59.140 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:11:59.139995 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:12:09.277 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:12:09.276896 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:12:09.289 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:12:09.288745 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:12:19.459 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:12:19.459353 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:12:19.469 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:12:19.469619 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:12:29.553 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:12:29.552634 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:12:29.564 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:12:29.564270 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:12:39.671 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:12:39.670458 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:12:39.683 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:12:39.683646 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:12:49.745 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:12:49.745170 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:12:49.758 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:12:49.757616 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:12:59.859 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:12:59.859177 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:12:59.873 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:12:59.872605 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:13:09.979 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:13:09.978595 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:13:10.003 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:13:10.003196 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:13:20.109 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:13:20.108661 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:13:20.124 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:13:20.123760 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:13:30.223 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:13:30.222941 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:13:30.235 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:13:30.235673 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:13:40.309 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:13:40.309022 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:13:40.326 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:13:40.325650 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:13:50.516 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:13:50.515945 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:13:50.526 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:13:50.525758 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:14:00.715 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:14:00.714517 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:14:00.730 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:14:00.727007 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:14:10.796 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:14:10.796281 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:14:10.811 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:14:10.810708 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:14:20.882 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:14:20.881707 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:14:20.896 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:14:20.896355 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:14:30.963 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:14:30.962601 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:14:30.979 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:14:30.978689 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:14:41.085 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:14:41.084685 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:14:41.102 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:14:41.101368 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:14:43.809 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:14:43.809026 I | op-mon: parsing mon endpoints: a=10.102.133.199:6789,b=10.107.147.59:6789,c=10.96.208.36:6789 2022-04-02T04:14:44.451 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:14:44.450532 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-04-02T04:14:44.437+0000 7f0f6f6c1380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-04-02T04:14:44.451 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02T04:14:44.437+0000 7f0f6f6c1380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-04-02T04:14:44.452 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02T04:14:44.437+0000 7f0f6f6c1380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-04-02T04:14:44.452 INFO:tasks.rook.operator.smithi029.stdout:couldn't init storage provider. : exit status 5 2022-04-02T04:14:51.246 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:14:51.245837 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:14:51.269 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:14:51.268777 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:15:01.336 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:15:01.336207 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:15:01.382 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:15:01.381655 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:15:11.484 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:15:11.483658 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:15:11.496 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:15:11.496239 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:15:21.559 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:15:21.559133 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:15:21.573 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:15:21.573154 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:15:31.664 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:15:31.663881 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:15:31.676 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:15:31.676107 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:15:41.781 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:15:41.780987 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:15:41.800 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:15:41.800355 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:15:51.876 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:15:51.875790 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:15:51.892 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:15:51.892541 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:16:02.003 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:16:02.002642 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:16:02.016 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:16:02.015816 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:16:12.120 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:16:12.119585 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:16:12.133 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:16:12.133431 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:16:22.242 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:16:22.241665 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:16:22.255 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:16:22.255021 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:16:32.367 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:16:32.366380 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:16:32.380 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:16:32.380112 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:16:42.486 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:16:42.485771 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:16:42.499 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:16:42.499541 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:16:52.570 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:16:52.569888 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:16:52.586 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:16:52.586135 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:17:02.655 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:17:02.655264 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:17:02.667 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:17:02.666833 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:17:12.759 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:17:12.759039 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:17:12.772 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:17:12.771867 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:17:22.862 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:17:22.862013 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:17:22.910 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:17:22.908860 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:17:33.041 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:17:33.040617 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:17:33.052 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:17:33.051573 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:17:43.146 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:17:43.145260 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:17:43.159 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:17:43.158958 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:17:53.234 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:17:53.233296 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:17:53.249 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:17:53.249079 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:18:03.357 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:18:03.357075 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:18:03.371 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:18:03.370694 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:18:13.444 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:18:13.443464 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:18:13.457 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:18:13.457211 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:18:23.528 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:18:23.527548 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:18:23.541 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:18:23.541530 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:18:33.616 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:18:33.615868 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:18:33.630 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:18:33.629860 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:18:43.884 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:18:43.883526 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:18:43.896 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:18:43.896288 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:18:54.016 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:18:54.015328 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:18:54.026 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:18:54.026501 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:19:04.133 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:19:04.133329 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:19:04.147 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:19:04.146461 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:19:14.217 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:19:14.216440 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:19:14.269 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:19:14.268296 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:19:24.388 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:19:24.387672 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:19:24.402 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:19:24.402382 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:19:34.477 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:19:34.476558 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:19:34.544 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:19:34.542639 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:19:44.659 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:19:44.658121 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:19:44.673 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:19:44.672290 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:19:54.774 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:19:54.773372 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:19:54.787 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:19:54.786410 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:20:04.925 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:20:04.924549 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:20:04.939 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:20:04.939063 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:20:15.053 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:20:15.046481 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:20:15.059 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:20:15.059133 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:20:25.144 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:20:25.143320 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:20:25.229 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:20:25.228756 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:20:35.341 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:20:35.340288 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:20:35.354 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:20:35.353991 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:20:45.479 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:20:45.478323 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:20:45.493 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:20:45.492893 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:20:55.562 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:20:55.562124 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:20:55.576 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:20:55.576235 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:21:05.654 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:21:05.653453 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:21:05.669 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:21:05.667801 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:21:15.746 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:21:15.745958 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:21:15.790 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:21:15.789441 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:21:25.900 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:21:25.899176 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:21:25.916 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:21:25.915532 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:21:35.980 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:21:35.979234 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:21:36.020 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:21:36.019467 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:21:46.082 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:21:46.081176 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:21:46.096 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:21:46.096144 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:21:56.206 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:21:56.205794 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:21:56.221 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:21:56.221125 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:22:06.344 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:22:06.343186 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:22:06.358 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:22:06.357814 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:22:16.427 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:22:16.426787 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:22:16.440 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:22:16.439356 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:22:26.571 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:22:26.570873 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:22:26.584 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:22:26.583760 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:22:36.687 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:22:36.686687 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:22:36.699 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:22:36.698730 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:22:46.780 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:22:46.779456 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:22:46.793 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:22:46.792454 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:22:56.890 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:22:56.889168 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:22:56.906 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:22:56.905879 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:23:07.010 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:23:07.009445 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:23:07.023 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:23:07.023013 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:23:17.188 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:23:17.186560 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:23:17.199 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:23:17.199109 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:23:27.327 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:23:27.326242 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:23:27.340 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:23:27.340142 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:23:37.451 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:23:37.449838 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:23:37.464 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:23:37.464147 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:23:47.599 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:23:47.598149 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:23:47.613 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:23:47.612829 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:23:57.679 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:23:57.678694 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:23:57.734 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:23:57.733182 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:24:07.842 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:24:07.841569 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:24:07.853 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:24:07.852932 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:24:17.924 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:24:17.923567 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:24:17.939 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:24:17.938791 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:24:28.047 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:24:28.046619 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:24:28.060 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:24:28.060094 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:24:38.156 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:24:38.155562 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:24:38.171 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:24:38.170578 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:24:48.251 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:24:48.250976 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:24:48.265 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:24:48.264933 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:24:58.372 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:24:58.371045 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:24:58.385 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:24:58.384978 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:25:08.460 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:25:08.459331 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:25:08.472 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:25:08.471372 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:25:18.580 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:25:18.579380 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:25:18.593 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:25:18.592483 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:25:28.668 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:25:28.667704 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:25:28.682 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:25:28.681957 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:25:38.840 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:25:38.839061 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:25:38.851 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:25:38.850868 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:25:48.930 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:25:48.929244 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:25:48.944 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:25:48.943728 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:25:59.041 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:25:59.040157 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:25:59.052 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:25:59.050992 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:26:09.125 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:26:09.124373 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:26:09.137 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:26:09.136506 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:26:19.251 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:26:19.249849 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:26:19.262 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:26:19.262132 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:26:29.376 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:26:29.375443 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:26:29.390 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:26:29.389310 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:26:39.508 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:26:39.507523 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:26:39.524 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:26:39.523264 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:26:49.589 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:26:49.588902 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:26:49.665 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:26:49.664318 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:26:59.812 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:26:59.811764 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:26:59.825 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:26:59.824378 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:27:09.903 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:27:09.902535 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:27:09.946 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:27:09.945844 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:27:20.085 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:27:20.084562 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:27:20.102 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:27:20.101412 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:27:30.178 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:27:30.177052 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:27:30.190 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:27:30.189624 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:27:40.253 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:27:40.252982 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:27:40.287 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:27:40.267655 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:27:50.358 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:27:50.357169 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:27:50.371 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:27:50.370386 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:28:00.501 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:28:00.500386 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:28:00.511 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:28:00.511171 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:28:10.591 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:28:10.590812 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:28:10.659 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:28:10.658659 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:28:20.730 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:28:20.729943 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:28:20.740 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:28:20.740172 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:28:30.809 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:28:30.808336 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:28:30.829 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:28:30.826401 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:28:40.935 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:28:40.934410 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:28:40.950 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:28:40.949475 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:28:51.020 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:28:51.019130 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:28:51.045 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:28:51.043416 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:29:01.167 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:29:01.166237 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:29:01.179 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:29:01.179009 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:29:11.252 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:29:11.251007 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:29:11.305 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:29:11.304890 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:29:21.410 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:29:21.409612 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:29:21.424 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:29:21.424341 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:29:31.531 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:29:31.530238 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:29:31.587 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:29:31.586383 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:29:41.662 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:29:41.661156 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:29:41.675 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:29:41.674017 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:29:51.787 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:29:51.785845 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:29:51.800 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:29:51.799395 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:30:01.909 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:30:01.908321 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:30:01.922 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:30:01.920244 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:30:12.022 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:30:12.021480 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:30:12.034 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:30:12.034374 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:30:22.140 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:30:22.139604 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:30:22.154 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:30:22.154039 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:30:32.249 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:30:32.248797 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:30:32.263 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:30:32.263293 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:30:42.366 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:30:42.365093 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:30:42.377 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:30:42.377220 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:30:52.518 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:30:52.516540 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:30:52.531 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:30:52.530693 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:31:02.658 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:31:02.658130 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:31:02.671 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:31:02.670550 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:31:12.740 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:31:12.739510 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:31:12.752 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:31:12.751916 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:31:22.908 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:31:22.907162 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:31:22.921 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:31:22.921440 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:31:24.460 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:31:24.459393 I | op-mon: parsing mon endpoints: a=10.102.133.199:6789,b=10.107.147.59:6789,c=10.96.208.36:6789 2022-04-02T04:31:25.108 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:31:25.107121 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-04-02T04:31:25.094+0000 7fd488a14380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-04-02T04:31:25.108 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02T04:31:25.094+0000 7fd488a14380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-04-02T04:31:25.109 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02T04:31:25.094+0000 7fd488a14380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-04-02T04:31:25.109 INFO:tasks.rook.operator.smithi029.stdout:couldn't init storage provider. : exit status 5 2022-04-02T04:31:33.033 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:31:33.032477 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:31:33.048 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:31:33.047820 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:31:43.190 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:31:43.189878 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:31:43.201 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:31:43.201381 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:31:53.304 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:31:53.303426 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:31:53.315 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:31:53.315372 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:32:03.423 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:32:03.422595 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:32:03.446 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:32:03.440560 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:32:13.543 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:32:13.542619 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:32:13.557 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:32:13.556780 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:32:23.663 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:32:23.662818 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:32:23.677 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:32:23.677078 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:32:33.742 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:32:33.741308 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:32:33.754 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:32:33.753834 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:32:43.832 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:32:43.831719 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:32:43.846 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:32:43.846288 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:32:53.988 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:32:53.987905 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:32:54.003 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:32:54.003106 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:33:04.105 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:33:04.105224 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:33:04.114 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:33:04.114394 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:33:14.214 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:33:14.213506 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:33:14.239 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:33:14.238661 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:33:24.365 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:33:24.364147 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:33:24.380 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:33:24.380098 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:33:34.442 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:33:34.441928 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:33:34.456 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:33:34.455542 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:33:44.537 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:33:44.536297 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:33:44.549 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:33:44.549089 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:33:54.623 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:33:54.622880 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:33:54.663 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:33:54.663219 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:34:04.778 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:34:04.777871 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:34:04.793 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:34:04.792530 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:34:14.912 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:34:14.911158 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:34:14.926 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:34:14.926141 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:34:25.006 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:34:25.005067 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:34:25.022 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:34:25.020017 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:34:35.095 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:34:35.094548 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:34:35.105 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:34:35.104960 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:34:45.172 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:34:45.171551 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:34:45.184 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:34:45.183500 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:34:55.313 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:34:55.312455 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:34:55.325 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:34:55.324765 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:35:05.399 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:35:05.399174 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:35:05.415 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:35:05.414280 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:35:15.535 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:35:15.534040 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:35:15.547 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:35:15.547080 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:35:25.620 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:35:25.620190 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:35:25.635 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:35:25.635119 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:35:35.705 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:35:35.704870 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:35:35.717 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:35:35.716879 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:35:45.819 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:35:45.818916 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:35:45.833 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:35:45.833532 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:35:55.929 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:35:55.928303 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:35:55.945 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:35:55.944734 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:36:06.075 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:36:06.074916 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:36:06.091 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:36:06.090970 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:36:16.194 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:36:16.194324 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:36:16.208 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:36:16.208412 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:36:26.327 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:36:26.326696 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:36:26.341 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:36:26.341397 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:36:36.469 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:36:36.469015 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:36:36.484 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:36:36.484377 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:36:46.567 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:36:46.567133 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:36:46.582 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:36:46.581973 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:36:56.669 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:36:56.668742 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:36:56.706 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:36:56.705872 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:37:06.786 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:37:06.785615 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:37:06.801 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:37:06.800799 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:37:16.905 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:37:16.904713 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:37:16.917 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:37:16.916646 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:37:26.988 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:37:26.987759 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:37:26.998 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:37:26.998111 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:37:37.072 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:37:37.071315 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:37:37.084 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:37:37.084383 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:37:47.155 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:37:47.154478 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:37:47.167 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:37:47.167528 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:37:57.244 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:37:57.243681 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:37:57.256 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:37:57.256106 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:38:07.370 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:38:07.369443 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:38:07.408 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:38:07.408018 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:38:17.511 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:38:17.510856 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:38:17.527 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:38:17.526936 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:38:27.599 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:38:27.599232 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:38:27.615 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:38:27.612576 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:38:37.671 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:38:37.668091 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:38:37.681 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:38:37.680509 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:38:47.824 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:38:47.823817 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:38:47.836 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:38:47.836248 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:38:57.986 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:38:57.986070 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:38:58.001 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:38:58.001360 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:39:08.067 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:39:08.067470 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:39:08.079 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:39:08.079439 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:39:18.191 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:39:18.190339 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:39:18.202 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:39:18.202563 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:39:28.312 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:39:28.311994 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:39:28.324 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:39:28.324263 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:39:38.440 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:39:38.439828 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:39:38.454 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:39:38.453761 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:39:48.536 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:39:48.535715 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:39:48.551 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:39:48.550580 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:39:58.654 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:39:58.653824 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:39:58.670 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:39:58.669351 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:40:08.747 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:40:08.747280 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:40:08.760 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:40:08.760379 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:40:18.831 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:40:18.830694 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:40:18.845 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:40:18.845206 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:40:28.923 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:40:28.923531 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:40:28.936 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:40:28.936228 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:40:39.050 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:40:39.050282 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:40:39.069 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:40:39.065732 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:40:49.203 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:40:49.202879 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:40:49.217 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:40:49.217677 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:40:59.333 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:40:59.332447 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:40:59.345 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:40:59.345769 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:41:09.479 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:41:09.478679 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:41:09.492 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:41:09.492875 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:41:19.588 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:41:19.588354 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:41:19.638 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:41:19.638090 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:41:29.743 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:41:29.743273 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:41:29.759 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:41:29.759581 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:41:39.901 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:41:39.901093 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:41:39.916 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:41:39.916393 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:41:50.024 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:41:50.023748 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:41:50.036 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:41:50.036982 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:42:00.118 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:42:00.117975 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:42:00.133 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:42:00.133355 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:42:10.196 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:42:10.196442 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:42:10.211 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:42:10.211754 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:42:20.277 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:42:20.277355 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:42:20.296 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:42:20.296604 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:42:30.367 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:42:30.367018 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:42:30.436 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:42:30.435860 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:42:40.511 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:42:40.511531 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:42:40.524 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:42:40.524732 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:42:50.640 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:42:50.640393 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:42:50.654 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:42:50.654021 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:43:00.725 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:43:00.725144 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:43:00.736 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:43:00.736346 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:43:10.802 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:43:10.802068 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:43:10.816 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:43:10.816597 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:43:21.072 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:43:21.071887 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:43:21.087 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:43:21.087359 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:43:31.215 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:43:31.214616 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:43:31.227 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:43:31.227257 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:43:41.362 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:43:41.361923 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:43:41.378 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:43:41.377435 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:43:51.447 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:43:51.447110 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:43:51.459 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:43:51.459640 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:44:01.579 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:44:01.579499 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:44:01.592 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:44:01.593060 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:44:11.707 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:44:11.706996 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:44:11.717 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:44:11.717530 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:44:21.788 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:44:21.787706 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:44:21.801 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:44:21.801556 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:44:31.872 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:44:31.872094 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:44:31.889 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:44:31.889986 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:44:41.990 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:44:41.990516 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:44:42.003 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:44:42.003384 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:44:52.073 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:44:52.073207 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:44:52.123 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:44:52.123753 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:45:02.192 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:45:02.191710 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:45:02.239 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:45:02.239373 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:45:12.350 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:45:12.349680 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:45:12.364 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:45:12.365086 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:45:22.471 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:45:22.470987 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:45:22.484 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:45:22.484255 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:45:32.558 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:45:32.558129 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:45:32.573 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:45:32.573026 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:45:42.676 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:45:42.675903 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:45:42.689 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:45:42.689705 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:45:52.767 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:45:52.767078 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:45:52.781 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:45:52.781875 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:46:02.843 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:46:02.843930 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:46:02.856 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:46:02.856743 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:46:12.987 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:46:12.987366 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:46:13.005 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:46:13.004955 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:46:23.083 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:46:23.083086 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:46:23.097 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:46:23.097065 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:46:33.200 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:46:33.200390 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:46:33.214 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:46:33.213974 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:46:43.339 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:46:43.339983 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:46:43.353 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:46:43.353778 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:46:53.463 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:46:53.463927 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:46:53.480 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:46:53.480724 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:47:03.592 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:47:03.592040 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:47:03.607 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:47:03.607689 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:47:13.688 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:47:13.688113 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:47:13.754 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:47:13.754573 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:47:23.823 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:47:23.823528 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:47:23.838 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:47:23.838307 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:47:33.912 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:47:33.912656 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:47:33.928 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:47:33.928630 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:47:44.036 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:47:44.036202 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:47:44.051 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:47:44.051367 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:47:54.126 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:47:54.126602 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:47:54.213 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:47:54.213414 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:48:04.365 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:48:04.365713 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:48:04.380 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:48:04.381072 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:48:05.115 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:48:05.115820 I | op-mon: parsing mon endpoints: a=10.102.133.199:6789,b=10.107.147.59:6789,c=10.96.208.36:6789 2022-04-02T04:48:05.795 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:48:05.794849 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-04-02T04:48:05.784+0000 7f86d14c8380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-04-02T04:48:05.795 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02T04:48:05.784+0000 7f86d14c8380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-04-02T04:48:05.795 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02T04:48:05.784+0000 7f86d14c8380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-04-02T04:48:05.795 INFO:tasks.rook.operator.smithi029.stdout:couldn't init storage provider. : exit status 5 2022-04-02T04:48:14.515 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:48:14.515830 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:48:14.529 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:48:14.530236 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:48:24.632 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:48:24.632737 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:48:24.644 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:48:24.644322 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:48:34.725 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:48:34.725088 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:48:34.747 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:48:34.743335 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:48:44.816 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:48:44.816202 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:48:44.829 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:48:44.830017 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:48:54.930 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:48:54.930060 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:48:54.944 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:48:54.944391 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:49:05.019 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:49:05.019783 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:49:05.034 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:49:05.034990 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:49:15.109 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:49:15.109094 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:49:15.122 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:49:15.122769 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:49:25.226 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:49:25.226823 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:49:25.239 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:49:25.240270 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:49:35.317 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:49:35.317785 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:49:35.332 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:49:35.332849 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:49:45.474 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:49:45.474783 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:49:45.487 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:49:45.487327 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:49:55.560 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:49:55.560321 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:49:55.658 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:49:55.658780 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:50:05.733 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:50:05.733841 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:50:05.780 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:50:05.779760 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:50:15.891 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:50:15.891719 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:50:15.907 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:50:15.907982 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:50:25.982 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:50:25.982484 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:50:25.995 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:50:25.996106 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:50:36.222 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:50:36.222112 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:50:36.234 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:50:36.234310 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:50:46.401 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:50:46.401175 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:50:46.415 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:50:46.415631 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:50:56.672 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:50:56.671579 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:50:56.682 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:50:56.682783 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:51:07.011 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:51:07.011671 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:51:07.026 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:51:07.026683 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:51:17.151 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:51:17.151641 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:51:17.164 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:51:17.164772 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:51:27.225 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:51:27.224393 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:51:27.313 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:51:27.313937 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:51:37.417 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:51:37.417486 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:51:37.432 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:51:37.432563 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:51:47.545 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:51:47.545768 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:51:47.557 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:51:47.558098 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:51:57.659 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:51:57.659381 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:51:57.674 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:51:57.674158 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:52:07.760 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:52:07.760499 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:52:07.838 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:52:07.838672 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:52:17.958 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:52:17.959081 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:52:17.970 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:52:17.970488 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:52:28.040 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:52:28.040070 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:52:28.053 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:52:28.053767 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:52:38.148 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:52:38.148184 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:52:38.161 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:52:38.161359 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:52:48.270 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:52:48.270555 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:52:48.285 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:52:48.285284 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:52:58.349 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:52:58.349024 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:52:58.363 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:52:58.363674 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:53:08.487 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:53:08.487800 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:53:08.502 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:53:08.503114 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:53:18.616 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:53:18.616241 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:53:18.630 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:53:18.630570 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:53:28.738 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:53:28.737952 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:53:28.751 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:53:28.751642 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:53:38.854 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:53:38.854573 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:53:38.868 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:53:38.868737 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:53:49.000 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:53:49.000466 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:53:49.014 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:53:49.014699 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:53:59.126 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:53:59.126397 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:53:59.139 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:53:59.139525 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:54:09.207 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:54:09.207458 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:54:09.219 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:54:09.219987 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:54:19.331 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:54:19.331678 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:54:19.344 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:54:19.344798 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:54:29.459 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:54:29.459382 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:54:29.474 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:54:29.474897 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:54:39.584 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:54:39.584484 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:54:39.598 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:54:39.598635 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:54:49.674 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:54:49.674650 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:54:49.753 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:54:49.754004 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:54:59.871 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:54:59.871529 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:54:59.889 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:54:59.889843 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:55:10.011 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:55:10.011744 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:55:10.025 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:55:10.025565 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:55:20.209 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:55:20.209351 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:55:20.223 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:55:20.223980 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:55:30.349 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:55:30.349220 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:55:30.384 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:55:30.384319 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:55:40.599 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:55:40.599409 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:55:40.612 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:55:40.612373 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:55:50.888 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:55:50.888851 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:55:50.904 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:55:50.904876 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:56:00.978 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:56:00.977866 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:56:00.991 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:56:00.991955 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:56:11.096 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:56:11.096547 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:56:11.111 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:56:11.111437 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:56:21.220 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:56:21.219911 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:56:21.235 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:56:21.235536 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:56:31.350 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:56:31.349906 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:56:31.364 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:56:31.364231 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:56:41.437 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:56:41.437726 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:56:41.451 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:56:41.452156 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:56:51.533 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:56:51.532817 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:56:51.547 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:56:51.548148 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:57:01.689 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:57:01.688831 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:57:01.701 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:57:01.701421 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:57:11.804 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:57:11.804436 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:57:11.818 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:57:11.818520 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:57:21.916 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:57:21.915996 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:57:21.931 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:57:21.931340 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:57:32.036 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:57:32.035908 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:57:32.051 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:57:32.051896 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:57:42.121 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:57:42.120731 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:57:42.139 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:57:42.139631 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:57:52.241 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:57:52.241291 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:57:52.253 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:57:52.253889 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:58:02.331 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:58:02.330979 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:58:02.344 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:58:02.344350 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:58:12.558 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:58:12.557584 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:58:12.568 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:58:12.568310 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:58:22.669 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:58:22.669254 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:58:22.683 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:58:22.683400 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:58:32.789 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:58:32.789728 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:58:32.802 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:58:32.802395 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:58:42.921 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:58:42.921000 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:58:42.934 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:58:42.934703 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:58:53.031 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:58:53.030596 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:58:53.080 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:58:53.080196 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:59:03.177 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:59:03.174801 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:59:03.187 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:59:03.187288 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:59:13.263 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:59:13.263523 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:59:13.320 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:59:13.320655 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:59:23.398 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:59:23.398489 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:59:23.471 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:59:23.471603 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:59:33.596 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:59:33.596819 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:59:33.610 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:59:33.610376 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:59:43.693 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:59:43.693496 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:59:43.706 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:59:43.706960 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:59:53.820 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:59:53.820006 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T04:59:53.835 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 04:59:53.835659 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:00:03.973 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:00:03.973249 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:00:03.987 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:00:03.987410 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:00:14.058 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:00:14.058769 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:00:14.103 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:00:14.103431 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:00:24.206 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:00:24.206297 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:00:24.219 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:00:24.219548 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:00:34.322 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:00:34.322253 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:00:34.335 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:00:34.335217 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:00:44.409 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:00:44.408639 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:00:44.458 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:00:44.458757 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:00:54.541 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:00:54.540887 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:00:54.555 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:00:54.555322 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:01:04.620 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:01:04.615581 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:01:04.673 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:01:04.672995 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:01:14.836 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:01:14.836558 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:01:14.846 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:01:14.846932 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:01:24.947 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:01:24.947445 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:01:24.959 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:01:24.959330 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:01:35.143 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:01:35.143081 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:01:35.152 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:01:35.152880 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:01:45.224 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:01:45.224077 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:01:45.278 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:01:45.278272 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:01:55.393 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:01:55.393241 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:01:55.407 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:01:55.407196 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:02:05.527 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:02:05.527279 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:02:05.543 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:02:05.544038 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:02:15.663 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:02:15.663470 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:02:15.677 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:02:15.677410 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:02:25.791 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:02:25.791480 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:02:25.805 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:02:25.805203 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:02:35.898 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:02:35.898466 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:02:35.912 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:02:35.912996 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:02:45.976 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:02:45.975681 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:02:45.994 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:02:45.994567 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:02:56.088 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:02:56.087435 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:02:56.096 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:02:56.096860 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:03:06.210 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:03:06.209839 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:03:06.224 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:03:06.224075 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:03:16.322 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:03:16.321816 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:03:16.335 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:03:16.335419 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:03:26.403 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:03:26.403434 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:03:26.416 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:03:26.416296 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:03:36.519 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:03:36.519185 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:03:36.531 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:03:36.531620 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:03:46.642 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:03:46.642626 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:03:46.658 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:03:46.658813 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:03:56.727 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:03:56.727032 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:03:56.739 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:03:56.739659 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:04:06.854 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:04:06.853948 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:04:06.868 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:04:06.868216 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:04:16.964 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:04:16.963904 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:04:16.978 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:04:16.978829 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:04:27.074 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:04:27.074635 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:04:27.087 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:04:27.087241 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:04:37.158 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:04:37.158188 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:04:37.245 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:04:37.245176 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:04:45.811 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:04:45.811569 I | op-mon: parsing mon endpoints: a=10.102.133.199:6789,b=10.107.147.59:6789,c=10.96.208.36:6789 2022-04-02T05:04:46.444 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:04:46.443849 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-04-02T05:04:46.433+0000 7fae1486b380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-04-02T05:04:46.444 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02T05:04:46.433+0000 7fae1486b380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-04-02T05:04:46.445 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02T05:04:46.433+0000 7fae1486b380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-04-02T05:04:46.445 INFO:tasks.rook.operator.smithi029.stdout:couldn't init storage provider. : exit status 5 2022-04-02T05:04:47.377 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:04:47.377624 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:04:47.391 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:04:47.391583 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:04:57.466 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:04:57.466302 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:04:57.519 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:04:57.519288 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:05:07.597 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:05:07.596843 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:05:07.643 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:05:07.643547 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:05:17.726 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:05:17.725358 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:05:17.739 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:05:17.739183 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:05:27.812 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:05:27.812129 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:05:27.825 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:05:27.825951 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:05:37.941 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:05:37.941198 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:05:37.953 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:05:37.953923 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:05:48.065 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:05:48.065645 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:05:48.078 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:05:48.079012 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:05:58.182 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:05:58.182428 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:05:58.197 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:05:58.198019 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:06:08.320 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:06:08.320424 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:06:08.336 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:06:08.336824 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:06:18.409 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:06:18.407794 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:06:18.451 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:06:18.451569 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:06:28.555 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:06:28.555329 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:06:28.587 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:06:28.587902 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:06:38.707 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:06:38.707709 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:06:38.720 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:06:38.720961 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:06:48.795 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:06:48.795393 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:06:49.245 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:06:48.807897 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:06:58.906 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:06:58.906005 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:06:58.920 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:06:58.921185 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:07:09.060 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:07:09.059800 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:07:09.075 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:07:09.076024 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:07:19.153 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:07:19.152795 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:07:19.165 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:07:19.166070 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:07:29.278 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:07:29.278800 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:07:29.289 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:07:29.290179 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:07:39.397 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:07:39.397181 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:07:39.411 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:07:39.410839 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:07:49.696 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:07:49.696503 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:07:49.714 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:07:49.714996 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:07:59.848 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:07:59.848068 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:07:59.861 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:07:59.861133 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:08:09.932 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:08:09.931742 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:08:09.944 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:08:09.944516 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:08:20.037 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:08:20.037305 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:08:20.052 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:08:20.052546 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:08:30.180 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:08:30.180169 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:08:30.195 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:08:30.195257 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:08:40.296 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:08:40.296205 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:08:40.310 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:08:40.310700 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:08:50.426 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:08:50.426131 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:08:50.440 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:08:50.440930 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:09:00.531 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:09:00.530595 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:09:00.543 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:09:00.543496 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:09:10.649 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:09:10.648948 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:09:10.662 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:09:10.662496 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:09:20.741 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:09:20.741482 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:09:20.789 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:09:20.789244 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:09:30.893 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:09:30.893603 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:09:30.909 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:09:30.909390 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:09:41.023 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:09:41.023243 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:09:41.037 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:09:41.037145 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:09:51.134 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:09:51.134438 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:09:51.151 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:09:51.149795 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:10:01.268 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:10:01.268287 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:10:01.286 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:10:01.286630 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:10:11.408 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:10:11.408181 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:10:11.421 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:10:11.421652 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:10:21.541 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:10:21.541134 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:10:21.554 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:10:21.554374 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:10:31.698 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:10:31.698542 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:10:31.712 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:10:31.712459 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:10:41.837 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:10:41.837489 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:10:41.851 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:10:41.851691 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:10:52.277 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:10:52.277229 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:10:52.291 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:10:52.291576 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:11:02.453 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:11:02.453593 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:11:02.468 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:11:02.468108 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:11:12.557 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:11:12.557308 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:11:12.575 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:11:12.575585 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:11:22.676 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:11:22.675598 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:11:22.689 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:11:22.689082 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:11:32.806 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:11:32.805562 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:11:32.821 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:11:32.821028 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:11:42.917 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:11:42.917233 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:11:42.931 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:11:42.931010 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:11:53.004 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:11:53.003668 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:11:53.016 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:11:53.016563 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:12:03.124 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:12:03.124686 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:12:03.137 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:12:03.137617 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:12:13.209 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:12:13.209443 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:12:13.219 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:12:13.219822 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:12:23.328 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:12:23.328255 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:12:23.338 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:12:23.338754 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:12:33.437 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:12:33.437338 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:12:33.452 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:12:33.452534 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:12:43.518 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:12:43.518069 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:12:43.531 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:12:43.531180 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:12:53.602 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:12:53.601912 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:12:53.615 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:12:53.615190 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:13:03.685 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:13:03.685301 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:13:03.697 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:13:03.697578 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:13:13.766 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:13:13.766327 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:13:13.842 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:13:13.842805 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:13:23.980 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:13:23.980517 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:13:23.991 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:13:23.991566 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:13:34.063 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:13:34.062960 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:13:34.075 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:13:34.075308 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:13:44.157 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:13:44.156995 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:13:44.171 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:13:44.171317 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:13:54.259 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:13:54.258633 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:13:54.273 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:13:54.273351 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:14:04.431 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:14:04.431075 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:14:04.446 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:14:04.446018 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:14:14.511 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:14:14.511351 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:14:14.561 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:14:14.561501 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:14:24.638 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:14:24.638242 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:14:24.651 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:14:24.651927 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:14:34.729 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:14:34.729309 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:14:34.823 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:14:34.744369 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:14:44.930 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:14:44.929767 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:14:44.969 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:14:44.969467 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:14:55.122 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:14:55.122444 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:14:55.136 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:14:55.136500 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:15:05.238 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:15:05.238417 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:15:05.253 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:15:05.253611 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:15:15.365 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:15:15.364818 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:15:15.380 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:15:15.380379 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:15:25.527 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:15:25.526255 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:15:25.539 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:15:25.539308 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:15:35.637 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:15:35.636890 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:15:35.651 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:15:35.651798 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:15:45.765 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:15:45.764875 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:15:45.780 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:15:45.780190 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:15:55.888 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:15:55.888080 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:15:55.904 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:15:55.904534 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:16:05.988 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:16:05.988248 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:16:06.002 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:16:06.002953 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:16:16.127 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:16:16.127226 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:16:16.137 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:16:16.137947 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:16:26.258 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:16:26.257794 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:16:26.271 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:16:26.271794 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:16:36.343 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:16:36.343443 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:16:36.358 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:16:36.357941 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:16:46.471 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:16:46.470770 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:16:46.483 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:16:46.483711 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:16:56.546 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:16:56.546574 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:16:56.560 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:16:56.560800 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:17:06.678 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:17:06.676648 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:17:06.689 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:17:06.689792 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:17:16.765 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:17:16.764712 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:17:16.779 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:17:16.779444 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:17:26.852 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:17:26.852236 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:17:26.896 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:17:26.895829 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:17:37.099 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:17:37.099181 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:17:37.112 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:17:37.112774 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:17:47.203 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:17:47.202913 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:17:47.217 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:17:47.217083 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:17:57.326 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:17:57.325916 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:17:57.342 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:17:57.342589 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:18:07.414 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:18:07.413861 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:18:07.426 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:18:07.426753 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:18:17.499 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:18:17.498735 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:18:17.512 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:18:17.512903 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:18:27.582 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:18:27.581542 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:18:27.596 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:18:27.596355 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:18:37.707 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:18:37.707311 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:18:37.722 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:18:37.722879 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:18:47.795 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:18:47.794993 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:18:47.811 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:18:47.811879 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:18:57.886 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:18:57.886022 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:18:57.903 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:18:57.902790 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:19:08.008 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:19:08.008176 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:19:08.022 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:19:08.022664 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:19:18.102 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:19:18.102127 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:19:18.117 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:19:18.117199 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:19:28.197 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:19:28.197712 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:19:28.211 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:19:28.211310 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:19:38.280 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:19:38.280514 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:19:38.338 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:19:38.338070 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:19:48.455 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:19:48.454966 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:19:48.468 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:19:48.468237 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:19:58.587 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:19:58.587449 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:19:58.603 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:19:58.603055 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:20:08.741 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:20:08.741595 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:20:08.752 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:20:08.752322 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:20:18.874 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:20:18.873912 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:20:18.887 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:20:18.887720 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:20:29.000 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:20:29.000332 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:20:29.009 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:20:29.010121 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:20:39.107 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:20:39.107088 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:20:39.177 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:20:39.177734 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:20:49.250 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:20:49.250991 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:20:49.264 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:20:49.264251 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:20:59.368 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:20:59.368214 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:20:59.382 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:20:59.382562 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:21:09.488 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:21:09.488283 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:21:09.501 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:21:09.501745 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:21:19.606 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:21:19.606581 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:21:19.619 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:21:19.619678 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:21:26.452 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:21:26.452281 I | op-mon: parsing mon endpoints: a=10.102.133.199:6789,b=10.107.147.59:6789,c=10.96.208.36:6789 2022-04-02T05:21:27.133 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:21:27.132658 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-04-02T05:21:27.122+0000 7f7c38aef380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-04-02T05:21:27.133 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02T05:21:27.122+0000 7f7c38aef380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-04-02T05:21:27.133 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02T05:21:27.122+0000 7f7c38aef380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-04-02T05:21:27.133 INFO:tasks.rook.operator.smithi029.stdout:couldn't init storage provider. : exit status 5 2022-04-02T05:21:29.697 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:21:29.696942 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:21:29.712 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:21:29.712841 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:21:39.785 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:21:39.785326 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:21:39.798 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:21:39.798820 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:21:49.896 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:21:49.896397 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:21:49.908 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:21:49.908278 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:22:00.064 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:22:00.064145 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:22:00.076 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:22:00.076631 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:22:10.168 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:22:10.167849 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:22:10.180 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:22:10.180330 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:22:20.273 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:22:20.272990 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:22:20.308 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:22:20.308670 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:22:30.409 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:22:30.409522 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:22:30.419 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:22:30.420186 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:22:40.525 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:22:40.525201 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:22:40.538 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:22:40.538334 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:22:50.626 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:22:50.626693 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:22:50.638 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:22:50.639174 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:23:00.713 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:23:00.713439 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:23:00.780 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:23:00.780421 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:23:10.877 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:23:10.877161 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:23:10.889 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:23:10.889570 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:23:20.986 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:23:20.987005 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:23:20.999 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:23:21.000347 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:23:31.074 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:23:31.074036 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:23:31.087 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:23:31.087616 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:23:41.202 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:23:41.202934 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:23:41.217 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:23:41.217549 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:23:51.286 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:23:51.286591 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:23:51.298 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:23:51.298777 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:24:01.374 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:24:01.373814 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:24:01.389 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:24:01.389463 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:24:11.461 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:24:11.461532 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:24:11.474 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:24:11.474665 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:24:21.594 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:24:21.594091 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:24:21.612 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:24:21.612546 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:24:31.750 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:24:31.750894 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:24:31.765 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:24:31.765910 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:24:41.843 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:24:41.843170 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:24:41.851 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:24:41.851937 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:24:51.921 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:24:51.921304 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:24:51.934 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:24:51.934706 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:25:02.037 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:25:02.036909 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:25:02.051 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:25:02.052023 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:25:12.127 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:25:12.127727 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:25:12.185 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:25:12.184632 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:25:22.249 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:25:22.249835 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:25:22.264 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:25:22.265080 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:25:32.362 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:25:32.362949 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:25:32.377 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:25:32.377658 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:25:42.462 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:25:42.462018 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:25:42.476 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:25:42.476814 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:25:52.547 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:25:52.547564 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:25:52.669 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:25:52.669567 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:26:02.769 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:26:02.769476 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:26:02.784 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:26:02.785313 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:26:12.888 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:26:12.888869 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:26:12.920 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:26:12.920832 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:26:22.993 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:26:22.993741 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:26:23.007 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:26:23.007438 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:26:33.094 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:26:33.094712 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:26:33.147 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:26:33.147908 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:26:43.256 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:26:43.257034 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:26:43.279 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:26:43.279517 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:26:53.338 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:26:53.335514 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:26:53.349 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:26:53.349958 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:27:03.479 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:27:03.479540 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:27:03.492 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:27:03.493233 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:27:13.562 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:27:13.562769 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:27:13.578 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:27:13.578695 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:27:23.691 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:27:23.691350 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:27:23.704 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:27:23.704769 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:27:33.839 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:27:33.839456 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:27:33.852 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:27:33.852894 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:27:43.961 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:27:43.961057 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:27:43.975 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:27:43.975651 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:27:54.106 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:27:54.107139 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:27:54.117 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:27:54.117958 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:28:04.256 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:28:04.256659 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:28:04.272 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:28:04.273175 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:28:14.381 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:28:14.381563 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:28:14.396 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:28:14.397370 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:28:24.523 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:28:24.523545 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:28:24.536 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:28:24.536589 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:28:34.627 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:28:34.627353 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:28:34.638 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:28:34.639119 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:28:44.776 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:28:44.776610 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:28:44.947 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:28:44.791018 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:28:54.890 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:28:54.890403 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:28:54.903 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:28:54.904504 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:29:05.015 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:29:05.014837 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:29:05.029 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:29:05.029826 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:29:15.138 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:29:15.138500 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:29:15.151 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:29:15.151763 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:29:25.218 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:29:25.218661 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:29:25.277 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:29:25.277887 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:29:35.351 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:29:35.351178 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:29:35.367 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:29:35.367892 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:29:45.508 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:29:45.508721 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:29:45.524 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:29:45.524911 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:29:55.639 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:29:55.639659 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:29:55.654 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:29:55.655241 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:30:05.731 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:30:05.731611 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:30:05.789 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:30:05.789413 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:30:15.905 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:30:15.904986 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:30:15.918 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:30:15.919503 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:30:26.003 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:30:26.003163 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:30:26.015 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:30:26.015513 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:30:36.185 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:30:36.185776 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:30:36.197 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:30:36.197536 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:30:46.272 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:30:46.272212 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:30:46.342 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:30:46.342563 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:30:56.470 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:30:56.470353 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:30:56.486 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:30:56.486393 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:31:06.576 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:31:06.576579 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:31:06.591 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:31:06.591687 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:31:16.714 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:31:16.714411 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:31:16.727 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:31:16.728165 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:31:26.829 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:31:26.828874 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:31:26.841 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:31:26.841942 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:31:36.951 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:31:36.951451 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:31:36.962 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:31:36.963321 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:31:47.065 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:31:47.065711 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:31:47.078 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:31:47.078981 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:31:57.174 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:31:57.174883 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:31:57.191 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:31:57.191692 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:32:07.314 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:32:07.314926 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:32:07.329 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:32:07.329710 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:32:17.456 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:32:17.456843 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:32:17.474 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:32:17.474422 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:32:27.587 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:32:27.587438 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:32:27.603 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:32:27.603282 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:32:37.674 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:32:37.674403 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:32:37.689 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:32:37.689561 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:32:47.764 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:32:47.764655 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:32:47.780 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:32:47.780108 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:32:57.879 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:32:57.879272 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:32:57.892 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:32:57.892973 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:33:08.001 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:33:08.000996 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:33:08.015 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:33:08.015371 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:33:18.110 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:33:18.110177 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:33:18.125 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:33:18.125460 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:33:28.190 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:33:28.190372 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:33:28.203 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:33:28.203234 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:33:38.299 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:33:38.299785 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:33:38.314 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:33:38.314636 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:33:48.421 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:33:48.421554 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:33:48.434 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:33:48.434724 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:33:58.545 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:33:58.545065 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:33:58.558 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:33:58.558759 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:34:08.659 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:34:08.659088 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:34:08.671 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:34:08.671672 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:34:18.748 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:34:18.748687 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:34:18.760 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:34:18.760818 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:34:28.829 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:34:28.829348 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:34:28.912 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:34:28.912959 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:34:38.992 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:34:38.992547 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:34:39.001 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:34:39.001420 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:34:49.111 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:34:49.110146 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:34:49.123 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:34:49.123456 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:34:59.195 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:34:59.195014 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:34:59.208 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:34:59.208443 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:35:09.302 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:35:09.301747 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:35:09.316 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:35:09.316008 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:35:19.393 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:35:19.391003 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:35:19.401 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:35:19.401648 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:35:29.500 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:35:29.500675 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:35:29.515 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:35:29.516013 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:35:39.620 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:35:39.620722 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:35:39.634 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:35:39.634882 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:35:49.715 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:35:49.715013 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:35:49.725 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:35:49.725225 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:35:59.805 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:35:59.805019 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:35:59.870 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:35:59.870209 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:36:09.997 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:36:09.997364 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:36:10.010 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:36:10.010925 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:36:20.206 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:36:20.206748 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:36:20.220 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:36:20.220547 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:36:30.367 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:36:30.367053 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:36:30.388 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:36:30.388668 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:36:40.461 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:36:40.461812 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:36:40.473 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:36:40.473832 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:36:50.581 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:36:50.581745 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:36:50.592 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:36:50.592718 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:37:00.706 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:37:00.706222 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:37:00.720 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:37:00.720480 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:37:10.797 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:37:10.797158 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:37:10.811 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:37:10.811349 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:37:20.917 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:37:20.917438 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:37:20.934 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:37:20.933274 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:37:31.002 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:37:31.002737 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:37:31.017 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:37:31.017937 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:37:41.168 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:37:41.168037 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:37:41.179 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:37:41.179852 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:37:51.325 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:37:51.325660 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:37:51.337 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:37:51.337322 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:38:01.699 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:38:01.697988 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:38:01.711 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:38:01.710927 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:38:07.139 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:38:07.139420 I | op-mon: parsing mon endpoints: a=10.102.133.199:6789,b=10.107.147.59:6789,c=10.96.208.36:6789 2022-04-02T05:38:07.837 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:38:07.835837 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-04-02T05:38:07.823+0000 7fac6e397380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-04-02T05:38:07.838 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02T05:38:07.823+0000 7fac6e397380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-04-02T05:38:07.838 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02T05:38:07.823+0000 7fac6e397380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-04-02T05:38:07.838 INFO:tasks.rook.operator.smithi029.stdout:couldn't init storage provider. : exit status 5 2022-04-02T05:38:11.966 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:38:11.965809 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:38:11.981 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:38:11.981925 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:38:22.080 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:38:22.080659 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:38:22.095 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:38:22.095097 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:38:32.372 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:38:32.371511 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:38:32.384 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:38:32.384240 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:38:42.656 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:38:42.656091 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:38:42.666 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:38:42.666598 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:38:52.757 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:38:52.757816 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:38:52.768 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:38:52.768948 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:39:03.086 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:39:03.086206 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:39:03.098 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:39:03.098725 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:39:13.362 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:39:13.361985 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:39:13.374 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:39:13.374610 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:39:23.506 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:39:23.506246 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:39:23.521 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:39:23.521680 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:39:33.772 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:39:33.772123 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:39:33.827 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:39:33.827520 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:39:44.082 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:39:44.082487 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:39:44.096 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:39:44.096016 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:39:54.165 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:39:54.165525 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:39:54.176 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:39:54.177211 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:40:04.245 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:40:04.244842 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:40:04.259 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:40:04.259248 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:40:14.330 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:40:14.330187 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:40:15.042 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:40:14.343552 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:40:24.416 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:40:24.416624 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:40:24.472 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:40:24.471989 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:40:34.616 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:40:34.616313 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:40:34.629 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:40:34.629369 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:40:44.766 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:40:44.766321 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:40:44.777 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:40:44.777400 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:40:54.854 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:40:54.853974 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:40:54.869 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:40:54.869301 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:41:04.958 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:41:04.958276 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:41:05.016 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:41:05.016363 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:41:15.123 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:41:15.123739 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:41:15.136 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:41:15.136765 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:41:25.255 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:41:25.254858 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:41:25.271 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:41:25.271961 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:41:35.341 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:41:35.340880 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:41:35.354 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:41:35.354552 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:41:45.435 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:41:45.435564 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:41:45.498 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:41:45.498647 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:41:55.574 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:41:55.574256 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:41:55.588 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:41:55.588491 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:42:05.713 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:42:05.713806 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:42:05.727 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:42:05.728037 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:42:15.814 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:42:15.814411 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:42:15.827 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:42:15.828131 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:42:25.913 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:42:25.912897 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:42:25.976 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:42:25.976536 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:42:36.044 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:42:36.044423 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:42:36.059 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:42:36.060135 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:42:46.135 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:42:46.134995 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:42:46.146 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:42:46.146506 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:42:56.213 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:42:56.213790 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:42:56.263 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:42:56.263574 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:43:06.408 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:43:06.407867 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:43:06.423 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:43:06.424106 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:43:16.560 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:43:16.559833 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:43:16.573 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:43:16.573245 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:43:26.735 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:43:26.735191 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:43:26.748 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:43:26.748303 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:43:36.870 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:43:36.870809 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:43:36.884 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:43:36.884383 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:43:46.990 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:43:46.990719 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:43:47.042 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:43:47.042271 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:43:57.176 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:43:57.176055 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:43:57.205 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:43:57.205425 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:44:07.317 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:44:07.312168 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:44:07.327 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:44:07.327676 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:44:17.405 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:44:17.405716 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:44:17.459 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:44:17.459292 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:44:27.573 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:44:27.573058 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:44:27.589 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:44:27.589808 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:44:37.661 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:44:37.661236 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:44:37.675 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:44:37.675897 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:44:47.755 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:44:47.755502 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:44:47.768 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:44:47.768247 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:44:57.884 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:44:57.884049 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:44:57.899 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:44:57.900257 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:45:08.007 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:45:08.007378 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:45:08.019 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:45:08.019420 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:45:18.122 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:45:18.121952 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:45:18.135 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:45:18.135980 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:45:28.289 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:45:28.288956 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:45:28.304 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:45:28.304837 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:45:38.406 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:45:38.406147 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:45:38.420 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:45:38.420637 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:45:48.546 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:45:48.546907 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:45:48.561 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:45:48.561892 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:45:58.637 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:45:58.637813 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:45:58.651 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:45:58.651666 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:46:08.775 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:46:08.775601 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:46:08.789 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:46:08.789231 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:46:18.863 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:46:18.863367 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:46:18.877 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:46:18.877065 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:46:29.026 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:46:29.023257 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:46:29.036 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:46:29.036578 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:46:39.140 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:46:39.140300 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:46:39.152 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:46:39.152566 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:46:49.359 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:46:49.359002 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:46:49.369 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:46:49.370140 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:46:59.455 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:46:59.455562 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:46:59.469 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:46:59.469374 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:47:09.694 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:47:09.694884 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:47:09.708 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:47:09.708673 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:47:19.801 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:47:19.801212 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:47:19.815 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:47:19.816269 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:47:29.926 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:47:29.926638 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:47:29.944 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:47:29.944191 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:47:40.015 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:47:40.015874 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:47:40.028 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:47:40.028852 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:47:50.122 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:47:50.122122 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:47:50.134 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:47:50.134723 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:48:00.223 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:48:00.223579 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:48:00.238 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:48:00.238913 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:48:10.338 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:48:10.338352 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:48:10.352 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:48:10.352283 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:48:20.464 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:48:20.464567 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:48:20.479 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:48:20.480180 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:48:30.549 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:48:30.549730 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:48:30.561 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:48:30.562343 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:48:40.673 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:48:40.672737 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:48:40.685 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:48:40.685558 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:48:50.784 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:48:50.784593 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:48:50.798 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:48:50.799399 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:49:00.868 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:49:00.868668 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:49:00.883 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:49:00.883707 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:49:10.997 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:49:10.997034 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:49:11.011 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:49:11.012194 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:49:21.169 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:49:21.169090 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:49:21.188 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:49:21.184069 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:49:31.283 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:49:31.282923 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:49:31.296 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:49:31.296392 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:49:41.366 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:49:41.366597 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:49:41.379 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:49:41.379891 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:49:51.476 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:49:51.476994 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:49:51.493 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:49:51.494101 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:50:01.594 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:50:01.594523 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:50:01.607 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:50:01.608020 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:50:11.703 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:50:11.703791 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:50:11.719 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:50:11.719710 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:50:21.794 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:50:21.795095 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:50:21.809 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:50:21.810295 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:50:31.928 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:50:31.928809 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:50:31.943 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:50:31.943319 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:50:42.034 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:50:42.034787 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:50:42.047 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:50:42.048172 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:50:52.156 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:50:52.156074 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:50:52.170 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:50:52.170371 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:51:02.287 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:51:02.287864 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:51:02.301 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:51:02.302153 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:51:12.373 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:51:12.373174 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:51:12.386 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:51:12.387082 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:51:22.456 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:51:22.456917 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:51:22.501 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:51:22.501230 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:51:32.605 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:51:32.605071 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:51:32.619 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:51:32.620172 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:51:42.731 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:51:42.731293 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:51:42.742 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:51:42.742891 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:51:52.816 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:51:52.816477 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:51:52.865 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:51:52.865466 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:52:02.970 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:52:02.970285 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:52:02.984 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:52:02.985327 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:52:13.061 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:52:13.061844 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:52:13.102 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:52:13.076714 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:52:23.171 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:52:23.171430 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:52:23.189 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:52:23.189967 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:52:33.294 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:52:33.294743 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:52:33.307 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:52:33.308187 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:52:43.415 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:52:43.415996 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:52:43.428 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:52:43.428520 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:52:53.577 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:52:53.577953 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:52:53.588 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:52:53.589024 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:53:03.784 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:53:03.784566 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:53:03.798 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:53:03.798496 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:53:13.915 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:53:13.915721 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:53:13.932 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:53:13.932582 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:53:24.039 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:53:24.039881 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:53:24.052 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:53:24.052585 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:53:34.209 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:53:34.209355 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:53:34.222 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:53:34.223330 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:53:44.324 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:53:44.324331 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:53:44.339 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:53:44.339887 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:53:54.422 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:53:54.422467 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:53:54.434 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:53:54.434630 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:54:04.535 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:54:04.535195 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:54:04.543 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:54:04.544237 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:54:14.656 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:54:14.656641 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:54:14.671 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:54:14.671668 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:54:24.779 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:54:24.779784 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:54:24.795 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:54:24.795318 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:54:34.942 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:54:34.942574 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:54:34.956 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:54:34.956537 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:54:45.037 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:54:45.037953 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:54:45.051 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:54:45.051826 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:54:47.845 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:54:47.845555 I | op-mon: parsing mon endpoints: a=10.102.133.199:6789,b=10.107.147.59:6789,c=10.96.208.36:6789 2022-04-02T05:54:48.496 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:54:48.496492 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-04-02T05:54:48.488+0000 7f2e67765380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-04-02T05:54:48.497 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02T05:54:48.488+0000 7f2e67765380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-04-02T05:54:48.497 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02T05:54:48.488+0000 7f2e67765380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-04-02T05:54:48.497 INFO:tasks.rook.operator.smithi029.stdout:couldn't init storage provider. : exit status 5 2022-04-02T05:54:55.144 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:54:55.144400 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:54:55.160 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:54:55.160716 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:55:05.243 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:55:05.243160 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:55:05.289 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:55:05.290029 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:55:15.404 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:55:15.404331 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:55:15.416 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:55:15.417182 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:55:25.541 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:55:25.541395 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:55:25.567 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:55:25.568075 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:55:35.718 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:55:35.718989 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:55:35.733 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:55:35.733592 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:55:45.853 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:55:45.853252 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:55:45.863 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:55:45.864460 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:55:55.987 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:55:55.987752 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:55:56.042 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:55:56.042748 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:56:06.118 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:56:06.118478 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:56:06.130 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:56:06.131077 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:56:16.259 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:56:16.259569 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:56:16.275 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:56:16.276428 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:56:26.387 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:56:26.387532 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:56:26.401 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:56:26.401927 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:56:36.514 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:56:36.514726 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:56:36.527 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:56:36.527674 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:56:46.622 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:56:46.622941 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:56:46.686 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:56:46.686860 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:56:56.791 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:56:56.791260 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:56:56.831 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:56:56.831941 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:57:06.962 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:57:06.962196 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:57:06.976 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:57:06.976945 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:57:17.069 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:57:17.069037 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:57:17.084 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:57:17.084679 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:57:27.206 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:57:27.206438 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:57:27.218 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:57:27.218744 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:57:37.298 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:57:37.298474 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:57:37.313 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:57:37.313516 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:57:47.415 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:57:47.415768 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:57:47.427 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:57:47.427961 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:57:57.547 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:57:57.547587 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:57:57.604 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:57:57.604509 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:58:07.703 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:58:07.703621 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:58:07.718 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:58:07.719052 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:58:17.827 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:58:17.827289 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:58:17.839 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:58:17.839844 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:58:27.903 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:58:27.903518 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:58:27.918 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:58:27.918384 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:58:38.026 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:58:38.026538 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:58:38.040 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:58:38.041286 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:58:48.112 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:58:48.112249 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:58:48.172 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:58:48.172971 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:58:58.304 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:58:58.304239 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:58:58.317 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:58:58.318503 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:59:08.421 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:59:08.421451 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:59:08.434 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:59:08.434552 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:59:18.551 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:59:18.546705 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:59:18.562 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:59:18.563021 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:59:28.726 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:59:28.726646 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:59:28.738 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:59:28.738835 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:59:38.859 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:59:38.860095 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:59:38.981 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:59:38.875992 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:59:48.940 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:59:48.940726 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:59:48.955 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:59:48.955504 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:59:59.076 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:59:59.077092 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T05:59:59.091 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 05:59:59.092208 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:00:09.196 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:00:09.196760 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:00:09.211 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:00:09.212203 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:00:19.291 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:00:19.290798 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:00:19.303 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:00:19.302812 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:00:29.368 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:00:29.368546 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:00:29.382 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:00:29.382518 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:00:39.453 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:00:39.454024 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:00:39.474 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:00:39.471433 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:00:49.635 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:00:49.635292 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:00:49.649 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:00:49.650303 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:00:59.716 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:00:59.716441 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:00:59.731 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:00:59.731529 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:01:09.841 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:01:09.841425 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:01:09.855 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:01:09.855495 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:01:19.928 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:01:19.928442 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:01:19.943 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:01:19.943426 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:01:30.050 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:01:30.050846 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:01:30.065 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:01:30.065851 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:01:40.147 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:01:40.148020 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:01:40.199 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:01:40.199975 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:01:50.327 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:01:50.327841 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:01:50.341 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:01:50.342627 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:02:00.448 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:02:00.448771 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:02:00.461 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:02:00.461586 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:02:10.534 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:02:10.534944 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:02:10.588 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:02:10.589003 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:02:20.686 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:02:20.686038 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:02:20.699 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:02:20.700559 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:02:30.776 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:02:30.776405 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:02:30.827 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:02:30.827408 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:02:40.944 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:02:40.944330 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:02:40.958 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:02:40.959041 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:02:51.034 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:02:51.034645 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:02:51.047 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:02:51.047937 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:03:01.119 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:03:01.119595 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:03:01.172 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:03:01.173213 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:03:11.242 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:03:11.243011 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:03:11.297 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:03:11.297591 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:03:21.400 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:03:21.400917 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:03:21.414 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:03:21.414832 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:03:31.491 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:03:31.491386 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:03:31.505 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:03:31.506377 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:03:41.664 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:03:41.664386 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:03:41.677 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:03:41.678113 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:03:51.796 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:03:51.796769 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:03:51.812 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:03:51.812558 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:04:01.879 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:04:01.879485 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:04:01.892 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:04:01.893152 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:04:12.005 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:04:12.006123 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:04:12.015 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:04:12.016459 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:04:22.117 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:04:22.117353 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:04:22.130 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:04:22.131306 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:04:32.240 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:04:32.240826 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:04:32.253 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:04:32.254629 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:04:42.378 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:04:42.378796 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:04:42.391 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:04:42.391687 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:04:52.527 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:04:52.527523 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:04:52.541 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:04:52.542124 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:05:02.708 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:05:02.709365 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:05:02.722 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:05:02.723352 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:05:12.836 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:05:12.836880 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:05:12.850 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:05:12.851379 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:05:23.102 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:05:23.102655 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:05:23.115 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:05:23.116563 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:05:33.524 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:05:33.524979 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:05:33.538 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:05:33.538961 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:05:43.658 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:05:43.658231 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:05:43.670 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:05:43.670755 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:05:53.931 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:05:53.931724 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:05:53.944 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:05:53.945688 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:06:04.228 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:06:04.229240 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:06:04.243 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:06:04.244341 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:06:14.328 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:06:14.328668 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:06:14.400 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:06:14.400939 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:06:24.615 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:06:24.616012 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:06:24.630 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:06:24.630813 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:06:34.867 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:06:34.867403 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:06:34.881 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:06:34.881670 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:06:44.956 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:06:44.956660 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:06:44.996 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:06:44.997428 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:06:55.255 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:06:55.256063 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:06:55.268 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:06:55.269231 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:07:05.513 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:07:05.513477 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:07:05.528 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:07:05.528885 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:07:15.681 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:07:15.681496 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:07:15.693 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:07:15.694296 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:07:25.997 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:07:25.998013 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:07:26.008 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:07:26.009587 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:07:36.258 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:07:36.258398 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:07:36.272 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:07:36.273284 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:07:46.391 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:07:46.391517 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:07:46.405 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:07:46.405549 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:07:56.479 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:07:56.479030 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:07:56.653 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:07:56.653412 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:08:06.771 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:08:06.772138 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:08:06.784 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:08:06.785152 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:08:16.853 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:08:16.854051 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:08:16.867 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:08:16.867974 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:08:26.942 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:08:26.942654 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:08:26.955 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:08:26.956067 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:08:37.028 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:08:37.028728 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:08:37.041 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:08:37.042755 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:08:47.161 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:08:47.162073 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:08:47.175 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:08:47.176254 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:08:57.246 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:08:57.246445 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:08:57.258 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:08:57.259390 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:09:07.329 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:09:07.330413 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:09:07.391 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:09:07.392033 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:09:17.458 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:09:17.458297 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:09:17.511 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:09:17.473911 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:09:27.548 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:09:27.548936 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:09:27.561 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:09:27.562444 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:09:37.634 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:09:37.634833 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:09:37.647 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:09:37.647917 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:09:47.746 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:09:47.746318 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:09:47.761 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:09:47.761761 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:09:57.859 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:09:57.859943 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:09:57.873 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:09:57.874139 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:10:07.947 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:10:07.947530 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:10:07.960 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:10:07.961435 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:10:18.072 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:10:18.073330 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:10:18.087 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:10:18.087864 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:10:28.161 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:10:28.161854 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:10:28.215 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:10:28.215869 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:10:38.320 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:10:38.321087 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:10:38.333 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:10:38.334161 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:10:48.437 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:10:48.438116 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:10:48.452 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:10:48.452671 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:10:58.540 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:10:58.541347 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:10:58.551 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:10:58.552028 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:11:08.657 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:11:08.657687 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:11:08.670 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:11:08.671168 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:11:18.802 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:11:18.802856 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:11:18.815 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:11:18.815730 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:11:28.507 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:11:28.507994 I | op-mon: parsing mon endpoints: a=10.102.133.199:6789,b=10.107.147.59:6789,c=10.96.208.36:6789 2022-04-02T06:11:28.942 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:11:28.943301 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:11:28.959 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:11:28.960740 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:11:29.193 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:11:29.193716 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-04-02T06:11:29.181+0000 7f36a6076380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-04-02T06:11:29.193 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02T06:11:29.181+0000 7f36a6076380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-04-02T06:11:29.194 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02T06:11:29.181+0000 7f36a6076380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-04-02T06:11:29.194 INFO:tasks.rook.operator.smithi029.stdout:couldn't init storage provider. : exit status 5 2022-04-02T06:11:39.033 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:11:39.034259 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:11:39.046 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:11:39.046828 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:11:49.157 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:11:49.157697 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:11:49.168 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:11:49.169359 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:11:59.308 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:11:59.309322 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:11:59.323 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:11:59.323587 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:12:09.440 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:12:09.441418 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:12:09.453 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:12:09.454688 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:12:19.604 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:12:19.604611 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:12:19.614 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:12:19.615248 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:12:29.722 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:12:29.722957 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:12:29.736 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:12:29.736753 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:12:39.809 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:12:39.810259 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:12:39.824 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:12:39.825357 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:12:50.015 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:12:50.015274 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:12:50.027 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:12:50.028771 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:13:00.106 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:13:00.106946 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:13:00.118 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:13:00.119193 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:13:10.245 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:13:10.245859 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:13:10.258 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:13:10.258855 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:13:20.462 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:13:20.462871 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:13:20.473 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:13:20.474023 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:13:30.608 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:13:30.608534 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:13:30.622 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:13:30.623119 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:13:40.731 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:13:40.731961 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:13:40.745 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:13:40.746376 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:13:50.856 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:13:50.856446 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:13:50.869 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:13:50.870609 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:14:00.991 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:14:00.991671 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:14:01.003 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:14:01.004811 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:14:11.064 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:14:11.064609 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:14:11.079 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:14:11.079501 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:14:21.151 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:14:21.151944 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:14:21.163 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:14:21.164393 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:14:31.236 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:14:31.236599 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:14:31.250 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:14:31.250852 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:14:41.446 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:14:41.441771 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:14:41.456 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:14:41.457495 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:14:51.563 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:14:51.563929 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:14:51.574 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:14:51.575383 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:15:01.687 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:15:01.688485 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:15:01.701 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:15:01.702495 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:15:11.903 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:15:11.903332 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:15:11.916 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:15:11.916928 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:15:22.035 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:15:22.035776 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:15:22.048 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:15:22.049247 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:15:32.168 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:15:32.168674 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:15:32.181 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:15:32.182387 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:15:42.284 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:15:42.284553 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:15:42.295 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:15:42.296342 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:15:52.361 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:15:52.361773 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:15:52.375 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:15:52.376327 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:16:02.446 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:16:02.446804 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:16:02.458 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:16:02.458967 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:16:12.546 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:16:12.546219 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:16:12.586 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:16:12.586466 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:16:22.671 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:16:22.671974 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:16:22.680 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:16:22.681115 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:16:32.743 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:16:32.741658 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:16:32.751 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:16:32.752432 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:16:42.861 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:16:42.861987 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:16:42.870 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:16:42.871157 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:16:52.992 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:16:52.992774 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:16:53.005 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:16:53.005957 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:17:03.107 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:17:03.107468 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:17:03.120 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:17:03.121165 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:17:13.241 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:17:13.241922 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:17:13.255 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:17:13.256282 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:17:23.368 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:17:23.369000 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:17:23.384 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:17:23.384794 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:17:33.484 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:17:33.484458 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:17:33.498 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:17:33.498777 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:17:43.602 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:17:43.602732 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:17:43.616 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:17:43.617168 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:17:53.719 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:17:53.719454 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:17:53.731 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:17:53.731844 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:18:03.834 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:18:03.834341 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:18:03.847 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:18:03.847735 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:18:13.915 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:18:13.916251 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:18:13.972 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:18:13.972235 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:18:24.039 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:18:24.040370 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:18:24.102 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:18:24.102561 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:18:34.225 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:18:34.225547 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:18:34.240 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:18:34.240678 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:18:44.362 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:18:44.362903 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:18:44.377 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:18:44.378538 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:18:54.481 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:18:54.482239 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:18:54.494 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:18:54.494583 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:19:04.615 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:19:04.616098 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:19:04.630 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:19:04.630671 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:19:14.732 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:19:14.733306 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:19:14.746 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:19:14.746962 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:19:24.825 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:19:24.825768 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:19:24.840 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:19:24.841286 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:19:34.899 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:19:34.899902 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:19:34.913 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:19:34.914036 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:19:45.027 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:19:45.027225 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:19:45.041 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:19:45.041815 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:19:55.112 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:19:55.112491 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:19:55.159 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:19:55.159694 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:20:05.266 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:20:05.266672 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:20:05.279 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:20:05.279698 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:20:15.385 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:20:15.385495 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:20:15.398 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:20:15.399065 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:20:25.468 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:20:25.468956 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:20:25.482 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:20:25.482993 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:20:35.616 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:20:35.616810 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:20:35.631 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:20:35.631509 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:20:45.849 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:20:45.849276 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:20:45.863 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:20:45.863730 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:20:55.969 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:20:55.970270 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:20:55.982 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:20:55.983204 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:21:06.152 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:21:06.153058 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:21:06.164 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:21:06.164677 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:21:16.274 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:21:16.274601 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:21:16.285 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:21:16.286068 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:21:26.375 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:21:26.375542 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:21:26.390 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:21:26.391121 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:21:36.468 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:21:36.468418 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:21:36.480 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:21:36.481486 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:21:46.628 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:21:46.628317 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:21:46.643 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:21:46.643791 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:21:56.755 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:21:56.755841 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:21:56.769 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:21:56.769807 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:22:06.916 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:22:06.916450 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:22:06.931 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:22:06.932191 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:22:17.025 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:22:17.025364 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:22:17.038 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:22:17.039505 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:22:27.136 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:22:27.136286 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:22:27.148 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:22:27.148935 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:22:37.213 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:22:37.213752 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:22:37.225 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:22:37.226403 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:22:47.324 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:22:47.324781 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:22:47.339 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:22:47.340307 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:22:57.475 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:22:57.475171 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:22:57.487 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:22:57.488106 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:23:07.621 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:23:07.621670 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:23:07.634 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:23:07.635125 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:23:17.737 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:23:17.737822 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:23:17.752 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:23:17.752743 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:23:27.850 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:23:27.850545 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:23:27.865 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:23:27.865974 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:23:37.979 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:23:37.979179 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:23:37.991 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:23:37.991953 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:23:48.105 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:23:48.105861 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:23:48.120 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:23:48.119944 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:23:58.186 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:23:58.187017 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:23:58.196 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:23:58.196769 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:24:08.318 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:24:08.318203 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:24:08.329 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:24:08.330064 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:24:18.408 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:24:18.408621 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:24:18.422 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:24:18.422576 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:24:28.526 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:24:28.526687 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:24:28.540 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:24:28.541449 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:24:38.686 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:24:38.686566 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:24:38.700 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:24:38.701395 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:24:48.863 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:24:48.863112 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:24:48.876 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:24:48.877561 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:24:58.953 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:24:58.953316 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:24:58.966 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:24:58.967122 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:25:09.103 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:25:09.103233 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:25:09.117 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:25:09.118354 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:25:19.249 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:25:19.249779 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:25:19.260 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:25:19.261236 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:25:29.361 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:25:29.361537 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:25:29.373 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:25:29.373578 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:25:39.482 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:25:39.482310 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:25:39.497 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:25:39.497739 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:25:49.610 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:25:49.611112 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:25:49.625 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:25:49.626063 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:25:59.719 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:25:59.719295 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:25:59.730 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:25:59.731341 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:26:09.817 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:26:09.817481 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:26:09.830 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:26:09.830853 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:26:19.911 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:26:19.911334 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:26:19.929 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:26:19.924795 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:26:29.995 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:26:29.995612 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:26:30.063 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:26:30.063457 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:26:40.133 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:26:40.134024 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:26:40.145 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:26:40.146430 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:26:50.226 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:26:50.222368 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:26:50.235 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:26:50.236249 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:27:00.319 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:27:00.319538 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:27:00.335 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:27:00.336474 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:27:10.410 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:27:10.410327 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:27:10.423 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:27:10.424558 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:27:20.527 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:27:20.527160 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:27:20.540 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:27:20.541070 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:27:30.634 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:27:30.634923 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:27:30.648 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:27:30.648821 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:27:40.714 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:27:40.714405 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:27:40.725 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:27:40.726298 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:27:50.958 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:27:50.957718 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:27:50.970 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:27:50.970456 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:28:01.082 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:28:01.082362 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:28:01.096 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:28:01.096602 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:28:09.201 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:28:09.201379 I | op-mon: parsing mon endpoints: a=10.102.133.199:6789,b=10.107.147.59:6789,c=10.96.208.36:6789 2022-04-02T06:28:09.848 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:28:09.847665 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-04-02T06:28:09.834+0000 7ff442b36380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-04-02T06:28:09.848 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02T06:28:09.834+0000 7ff442b36380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-04-02T06:28:09.848 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02T06:28:09.834+0000 7ff442b36380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-04-02T06:28:09.848 INFO:tasks.rook.operator.smithi029.stdout:couldn't init storage provider. : exit status 5 2022-04-02T06:28:11.208 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:28:11.208691 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:28:11.228 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:28:11.223470 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:28:21.355 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:28:21.356128 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:28:21.439 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:28:21.369929 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:28:31.479 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:28:31.480115 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:28:31.489 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:28:31.489986 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:28:41.562 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:28:41.562174 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:28:41.603 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:28:41.603623 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:28:51.678 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:28:51.677031 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:28:51.688 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:28:51.688929 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:29:01.788 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:29:01.788603 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:29:01.801 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:29:01.801659 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:29:11.912 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:29:11.913023 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:29:11.925 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:29:11.926109 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:29:21.989 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:29:21.989766 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:29:22.058 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:29:22.058444 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:29:32.171 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:29:32.171841 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:29:32.184 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:29:32.184714 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:29:42.292 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:29:42.292821 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:29:42.307 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:29:42.307720 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:29:52.409 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:29:52.408961 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:29:52.424 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:29:52.424630 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:30:02.539 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:30:02.539405 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:30:02.553 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:30:02.554044 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:30:12.630 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:30:12.629864 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:30:12.643 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:30:12.644134 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:30:22.761 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:30:22.761447 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:30:22.774 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:30:22.774819 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:30:32.840 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:30:32.841061 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:30:32.881 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:30:32.882191 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:30:42.997 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:30:42.997974 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:30:43.012 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:30:43.012419 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:30:53.145 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:30:53.145595 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:30:53.161 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:30:53.162649 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:31:03.287 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:31:03.287097 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:31:03.299 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:31:03.299562 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:31:13.423 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:31:13.423216 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:31:13.436 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:31:13.436764 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:31:23.554 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:31:23.554540 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:31:23.566 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:31:23.567185 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:31:33.690 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:31:33.690186 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:31:33.704 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:31:33.704638 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:31:43.791 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:31:43.791873 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:31:43.804 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:31:43.804821 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:31:53.869 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:31:53.869264 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:31:53.881 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:31:53.881779 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:32:03.960 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:32:03.960068 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:32:03.974 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:32:03.974430 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:32:14.045 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:32:14.045005 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:32:14.110 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:32:14.110642 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:32:24.310 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:32:24.310435 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:32:24.327 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:32:24.328052 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:32:34.459 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:32:34.459372 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:32:34.473 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:32:34.473819 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:32:44.594 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:32:44.594426 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:32:44.606 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:32:44.606639 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:32:54.706 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:32:54.706530 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:32:54.719 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:32:54.720084 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:33:04.825 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:33:04.825877 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:33:04.840 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:33:04.840984 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:33:14.909 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:33:14.909419 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:33:14.981 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:33:14.981947 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:33:25.077 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:33:25.077305 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:33:25.093 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:33:25.093244 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:33:35.196 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:33:35.195865 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:33:35.210 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:33:35.211035 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:33:45.302 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:33:45.302842 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:33:45.315 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:33:45.316147 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:33:55.399 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:33:55.399379 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:33:55.412 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:33:55.412658 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:34:05.483 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:34:05.483236 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:34:05.496 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:34:05.497201 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:34:15.613 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:34:15.613330 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:34:15.641 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:34:15.642240 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:34:25.708 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:34:25.708123 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:34:25.719 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:34:25.719905 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:34:35.830 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:34:35.829955 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:34:35.843 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:34:35.843708 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:34:45.941 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:34:45.941420 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:34:45.955 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:34:45.955566 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:34:56.055 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:34:56.055677 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:34:56.069 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:34:56.070019 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:35:06.182 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:35:06.182366 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:35:06.195 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:35:06.195900 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:35:16.266 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:35:16.266219 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:35:16.279 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:35:16.279368 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:35:26.343 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:35:26.343534 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:35:26.360 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:35:26.360887 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:35:36.427 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:35:36.427397 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:35:36.441 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:35:36.441322 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:35:46.550 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:35:46.550170 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:35:46.566 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:35:46.566331 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:35:56.656 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:35:56.656829 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:35:56.734 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:35:56.734363 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:36:06.841 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:36:06.840880 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:36:06.855 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:36:06.856221 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:36:16.982 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:36:16.982102 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:36:17.000 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:36:17.000922 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:36:27.122 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:36:27.122080 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:36:27.135 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:36:27.135561 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:36:37.225 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:36:37.225324 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:36:37.241 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:36:37.241330 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:36:47.310 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:36:47.310658 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:36:47.325 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:36:47.325406 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:36:57.408 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:36:57.408304 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:36:57.421 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:36:57.421984 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:37:07.531 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:37:07.531025 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:37:07.546 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:37:07.546625 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:37:17.643 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:37:17.643426 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:37:17.656 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:37:17.656808 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:37:27.728 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:37:27.729089 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:37:27.743 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:37:27.743979 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:37:37.836 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:37:37.837188 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:37:37.850 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:37:37.851040 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:37:47.921 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:37:47.921836 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:37:47.971 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:37:47.971072 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:37:58.087 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:37:58.087004 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:37:58.099 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:37:58.099550 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:38:08.196 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:38:08.196403 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:38:08.213 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:38:08.214134 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:38:18.317 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:38:18.317316 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:38:18.331 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:38:18.332288 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:38:28.405 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:38:28.405276 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:38:28.418 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:38:28.419036 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:38:38.523 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:38:38.523159 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:38:38.535 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:38:38.535346 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:38:48.619 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:38:48.619711 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:38:48.655 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:38:48.655122 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:38:58.725 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:38:58.724897 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:38:58.738 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:38:58.739058 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:39:08.843 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:39:08.842960 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:39:08.861 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:39:08.861893 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:39:18.930 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:39:18.929942 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:39:18.997 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:39:18.997005 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:39:29.073 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:39:29.073331 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:39:29.117 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:39:29.117704 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:39:39.196 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:39:39.196862 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:39:39.210 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:39:39.210725 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:39:49.276 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:39:49.276672 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:39:49.287 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:39:49.287852 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:39:59.383 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:39:59.383269 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:39:59.395 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:39:59.395849 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:40:09.504 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:40:09.504802 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:40:09.520 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:40:09.520570 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:40:19.592 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:40:19.591902 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:40:19.607 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:40:19.607996 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:40:29.679 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:40:29.679559 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:40:29.726 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:40:29.726428 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:40:39.800 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:40:39.799736 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:40:39.813 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:40:39.813818 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:40:49.919 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:40:49.919254 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:40:49.936 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:40:49.936893 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:41:00.004 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:41:00.003933 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:41:00.055 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:41:00.055796 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:41:10.174 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:41:10.173951 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:41:10.186 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:41:10.186507 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:41:20.339 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:41:20.339145 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:41:20.350 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:41:20.350265 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:41:30.486 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:41:30.486174 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:41:30.496 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:41:30.496937 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:41:40.564 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:41:40.562215 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:41:40.576 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:41:40.575607 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:41:50.677 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:41:50.677502 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:41:50.690 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:41:50.691248 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:42:00.764 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:42:00.764655 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:42:00.776 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:42:00.776660 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:42:10.846 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:42:10.846385 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:42:10.860 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:42:10.860371 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:42:20.963 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:42:20.963739 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:42:20.979 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:42:20.979417 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:42:31.085 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:42:31.085148 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:42:31.098 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:42:31.098770 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:42:41.196 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:42:41.196376 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:42:41.211 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:42:41.211381 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:42:51.287 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:42:51.287069 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:42:51.301 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:42:51.302227 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:43:01.437 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:43:01.437043 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:43:01.450 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:43:01.450859 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:43:11.527 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:43:11.527118 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:43:11.572 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:43:11.573014 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:43:21.631 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:43:21.631590 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:43:21.643 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:43:21.643444 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:43:31.749 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:43:31.749270 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:43:31.764 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:43:31.764026 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:43:41.845 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:43:41.844699 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:43:41.857 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:43:41.857648 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:43:51.963 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:43:51.963325 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:43:51.978 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:43:51.978985 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:44:02.113 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:44:02.113225 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:44:02.126 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:44:02.125837 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:44:12.235 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:44:12.235186 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:44:12.248 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:44:12.248750 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:44:22.382 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:44:22.382761 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:44:22.395 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:44:22.395402 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:44:32.501 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:44:32.501255 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:44:32.513 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:44:32.513585 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:44:42.656 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:44:42.656711 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:44:42.669 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:44:42.669442 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:44:49.857 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:44:49.857135 I | op-mon: parsing mon endpoints: a=10.102.133.199:6789,b=10.107.147.59:6789,c=10.96.208.36:6789 2022-04-02T06:44:50.550 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:44:50.550019 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-04-02T06:44:50.539+0000 7efea0f0c380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-04-02T06:44:50.550 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02T06:44:50.539+0000 7efea0f0c380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-04-02T06:44:50.551 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02T06:44:50.539+0000 7efea0f0c380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-04-02T06:44:50.551 INFO:tasks.rook.operator.smithi029.stdout:couldn't init storage provider. : exit status 5 2022-04-02T06:44:52.744 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:44:52.744506 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:44:52.758 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:44:52.758739 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:45:02.846 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:45:02.846053 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:45:02.860 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:45:02.861063 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:45:12.940 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:45:12.940330 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:45:12.949 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:45:12.950256 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:45:23.021 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:45:23.020657 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:45:23.035 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:45:23.035490 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:45:33.115 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:45:33.114785 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:45:33.131 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:45:33.126591 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:45:43.249 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:45:43.248757 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:45:43.261 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:45:43.261557 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:45:53.388 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:45:53.388228 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:45:53.398 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:45:53.398671 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:46:03.468 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:46:03.468107 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:46:03.482 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:46:03.482461 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:46:13.557 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:46:13.556756 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:46:13.569 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:46:13.569203 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:46:23.659 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:46:23.659869 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:46:23.674 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:46:23.674204 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:46:33.800 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:46:33.800552 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:46:33.814 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:46:33.814123 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:46:43.896 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:46:43.896108 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:46:43.909 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:46:43.909656 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:46:54.032 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:46:54.032193 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:46:54.067 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:46:54.067530 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:47:04.138 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:47:04.137747 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:47:04.178 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:47:04.178298 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:47:14.247 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:47:14.247350 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:47:14.261 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:47:14.261745 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:47:24.332 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:47:24.332333 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:47:24.344 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:47:24.344175 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:47:34.418 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:47:34.418652 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:47:34.491 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:47:34.491632 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:47:44.619 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:47:44.619231 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:47:44.634 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:47:44.634198 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:47:54.709 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:47:54.708648 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:47:54.879 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:47:54.879682 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:48:05.066 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:48:05.066604 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:48:05.080 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:48:05.080894 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:48:16.957 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:48:16.956823 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:48:16.970 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:48:16.970480 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:48:27.086 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:48:27.086048 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:48:27.128 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:48:27.128751 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:48:37.223 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:48:37.222937 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:48:37.236 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:48:37.236127 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:48:47.322 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:48:47.322568 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:48:47.332 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:48:47.332617 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:48:57.470 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:48:57.469837 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:48:57.489 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:48:57.486056 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:49:07.596 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:49:07.596055 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:49:07.610 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:49:07.609861 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:49:17.713 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:49:17.712849 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:49:17.727 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:49:17.727707 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:49:27.851 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:49:27.850522 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:49:27.864 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:49:27.864182 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:49:37.977 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:49:37.976398 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:49:37.989 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:49:37.989774 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:49:48.058 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:49:48.058214 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:49:48.144 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:49:48.144368 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:49:58.210 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:49:58.210247 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:49:58.224 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:49:58.224656 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:50:08.325 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:50:08.325318 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:50:08.338 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:50:08.338885 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:50:18.483 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:50:18.483589 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:50:18.496 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:50:18.496951 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:50:28.575 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:50:28.575414 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:50:28.589 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:50:28.589197 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:50:38.715 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:50:38.714781 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:50:38.730 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:50:38.730585 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:50:48.846 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:50:48.845832 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:50:48.861 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:50:48.861424 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:50:58.979 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:50:58.978852 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:50:58.994 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:50:58.994749 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:51:09.065 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:51:09.065554 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:51:09.083 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:51:09.083631 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:51:19.195 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:51:19.195107 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:51:19.208 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:51:19.208796 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:51:29.318 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:51:29.318760 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:51:29.332 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:51:29.332225 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:51:39.436 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:51:39.435959 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:51:39.461 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:51:39.452428 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:51:49.565 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:51:49.565043 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:51:49.578 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:51:49.578469 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:51:59.726 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:51:59.725941 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:51:59.742 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:51:59.742010 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:52:09.808 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:52:09.807825 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:52:09.822 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:52:09.822177 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:52:19.921 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:52:19.921260 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:52:19.936 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:52:19.936478 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:52:30.069 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:52:30.068635 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:52:30.089 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:52:30.089189 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:52:40.208 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:52:40.208444 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:52:40.220 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:52:40.220842 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:52:50.302 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:52:50.302572 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:52:50.341 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:52:50.341757 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:53:00.416 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:53:00.416192 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:53:00.428 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:53:00.428516 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:53:10.545 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:53:10.545188 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:53:10.560 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:53:10.560592 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:53:20.632 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:53:20.632588 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:53:20.646 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:53:20.646671 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:53:30.774 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:53:30.774522 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:53:30.788 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:53:30.788716 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:53:40.925 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:53:40.925337 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:53:40.942 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:53:40.942501 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:53:51.081 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:53:51.081439 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:53:51.096 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:53:51.096147 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:54:01.165 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:54:01.165547 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:54:01.177 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:54:01.177504 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:54:13.772 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:54:13.771880 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:54:13.783 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:54:13.783834 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:54:23.860 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:54:23.860115 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:54:23.874 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:54:23.874386 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:54:33.941 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:54:33.941395 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:54:33.956 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:54:33.956187 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:54:44.057 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:54:44.057210 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:54:44.069 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:54:44.069872 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:54:54.169 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:54:54.169232 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:54:54.184 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:54:54.184891 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:55:04.299 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:55:04.299168 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:55:04.310 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:55:04.309983 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:55:14.431 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:55:14.430986 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:55:14.448 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:55:14.448417 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:55:24.542 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:55:24.542482 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:55:24.556 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:55:24.556627 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:55:34.626 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:55:34.626478 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:55:34.637 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:55:34.637317 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:55:44.711 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:55:44.710633 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:55:44.725 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:55:44.724983 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:55:54.797 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:55:54.796586 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:55:54.855 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:55:54.855665 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:56:04.952 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:56:04.951910 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:56:04.973 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:56:04.973300 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:56:15.106 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:56:15.106044 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:56:15.118 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:56:15.117767 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:56:25.239 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:56:25.238619 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:56:25.255 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:56:25.255571 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:56:35.329 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:56:35.328698 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:56:35.343 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:56:35.343540 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:56:45.412 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:56:45.412285 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:56:45.425 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:56:45.424981 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:56:55.500 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:56:55.499685 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:56:55.515 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:56:55.515304 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:57:05.633 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:57:05.633481 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:57:05.648 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:57:05.648608 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:57:15.719 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:57:15.719259 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:57:15.734 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:57:15.734512 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:57:25.847 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:57:25.846797 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:57:25.862 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:57:25.862784 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:57:35.932 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:57:35.932385 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:57:35.946 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:57:35.945526 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:57:46.055 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:57:46.054941 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:57:46.069 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:57:46.068841 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:57:56.165 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:57:56.165418 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:57:56.178 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:57:56.178240 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:58:06.298 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:58:06.298596 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:58:06.309 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:58:06.309776 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:58:16.385 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:58:16.385225 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:58:16.399 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:58:16.398880 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:58:26.504 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:58:26.504188 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:58:26.527 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:58:26.522385 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:58:36.586 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:58:36.586427 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:58:36.644 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:58:36.644025 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:58:46.711 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:58:46.710748 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:58:46.724 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:58:46.723966 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:58:56.843 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:58:56.842371 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:58:56.855 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:58:56.855288 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:59:06.971 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:59:06.970854 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:59:06.985 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:59:06.984838 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:59:17.059 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:59:17.058865 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:59:17.071 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:59:17.071284 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:59:27.185 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:59:27.184324 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:59:27.198 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:59:27.198058 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:59:37.275 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:59:37.275408 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:59:37.288 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:59:37.288957 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:59:47.394 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:59:47.393503 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:59:47.406 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:59:47.406050 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:59:57.488 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:59:57.488499 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T06:59:57.502 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 06:59:57.501648 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T07:00:07.583 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 07:00:07.583359 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T07:00:07.592 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 07:00:07.592438 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T07:00:17.796 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 07:00:17.795431 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T07:00:17.807 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 07:00:17.807636 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T07:00:27.949 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 07:00:27.948549 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T07:00:27.962 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 07:00:27.961999 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T07:00:38.043 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 07:00:38.043162 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T07:00:38.059 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 07:00:38.057749 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T07:00:48.190 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 07:00:48.189625 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T07:00:48.203 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 07:00:48.203148 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T07:00:58.329 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 07:00:58.329633 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T07:00:58.341 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 07:00:58.341064 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T07:01:08.475 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 07:01:08.475328 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T07:01:08.491 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 07:01:08.490896 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T07:01:18.600 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 07:01:18.600186 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T07:01:18.614 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 07:01:18.614284 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T07:01:28.690 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 07:01:28.689582 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T07:01:28.702 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 07:01:28.702206 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T07:01:30.573 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 07:01:30.573374 I | op-mon: parsing mon endpoints: a=10.102.133.199:6789,b=10.107.147.59:6789,c=10.96.208.36:6789 2022-04-02T07:01:31.333 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 07:01:31.332362 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-04-02T07:01:31.324+0000 7fa42eecc380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-04-02T07:01:31.334 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02T07:01:31.324+0000 7fa42eecc380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-04-02T07:01:31.334 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02T07:01:31.324+0000 7fa42eecc380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-04-02T07:01:31.335 INFO:tasks.rook.operator.smithi029.stdout:couldn't init storage provider. : exit status 5 2022-04-02T07:01:38.802 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 07:01:38.801981 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T07:01:38.813 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 07:01:38.813292 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T07:01:48.958 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 07:01:48.957544 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T07:01:48.970 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 07:01:48.969955 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T07:01:59.104 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 07:01:59.103791 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T07:01:59.117 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 07:01:59.117575 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T07:02:09.251 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 07:02:09.251492 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T07:02:09.262 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 07:02:09.261953 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T07:02:19.345 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 07:02:19.345284 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T07:02:19.360 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 07:02:19.360077 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T07:02:29.468 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 07:02:29.468368 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T07:02:29.485 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 07:02:29.484953 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T07:02:39.577 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 07:02:39.574347 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T07:02:39.623 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 07:02:39.623298 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T07:02:49.726 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 07:02:49.726260 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T07:02:49.742 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 07:02:49.742225 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T07:02:59.829 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 07:02:59.829351 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T07:02:59.845 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 07:02:59.844904 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T07:03:09.947 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 07:03:09.947180 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T07:03:09.960 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 07:03:09.960259 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T07:03:20.083 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 07:03:20.082662 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T07:03:20.095 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 07:03:20.095198 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T07:03:30.225 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 07:03:30.224707 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T07:03:30.238 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 07:03:30.238732 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T07:03:40.337 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 07:03:40.336894 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T07:03:40.350 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 07:03:40.349905 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T07:03:50.424 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 07:03:50.424292 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T07:03:50.438 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 07:03:50.437909 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T07:04:00.505 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 07:04:00.505440 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T07:04:00.563 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 07:04:00.563059 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T07:04:10.663 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 07:04:10.663488 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T07:04:10.696 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 07:04:10.696291 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T07:04:20.816 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 07:04:20.815621 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T07:04:20.831 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 07:04:20.831815 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T07:04:30.943 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 07:04:30.942812 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T07:04:30.958 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 07:04:30.958394 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T07:04:41.051 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 07:04:41.051573 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T07:04:41.063 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 07:04:41.063579 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T07:04:51.130 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 07:04:51.130217 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T07:04:51.146 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 07:04:51.145900 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T07:05:01.209 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 07:05:01.209393 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T07:05:01.222 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 07:05:01.222132 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T07:05:11.322 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 07:05:11.322339 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T07:05:11.336 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 07:05:11.336556 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T07:05:21.474 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 07:05:21.473610 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T07:05:21.489 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 07:05:21.489357 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T07:05:31.623 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 07:05:31.622621 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T07:05:31.640 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 07:05:31.640175 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T07:05:41.735 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 07:05:41.734753 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T07:05:41.746 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 07:05:41.746580 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T07:05:51.845 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 07:05:51.845365 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T07:05:51.866 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 07:05:51.866133 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T07:06:01.940 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 07:06:01.939668 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T07:06:01.953 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 07:06:01.953144 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T07:06:12.107 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 07:06:12.106725 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T07:06:12.122 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 07:06:12.122346 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T07:06:22.503 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 07:06:22.502962 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T07:06:22.517 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 07:06:22.517028 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T07:06:32.641 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 07:06:32.640731 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T07:06:32.658 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 07:06:32.658589 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T07:06:42.733 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 07:06:42.732910 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T07:06:42.777 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 07:06:42.776894 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T07:06:52.847 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 07:06:52.846965 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T07:06:52.900 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 07:06:52.900543 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T07:07:03.005 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 07:07:03.004642 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T07:07:03.019 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 07:07:03.019326 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T07:07:13.135 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 07:07:13.135082 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T07:07:13.151 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 07:07:13.150964 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T07:07:23.256 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 07:07:23.255491 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T07:07:23.268 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 07:07:23.268049 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T07:07:33.392 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 07:07:33.392148 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T07:07:33.405 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 07:07:33.405129 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T07:07:43.559 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 07:07:43.559415 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T07:07:43.574 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 07:07:43.573548 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T07:07:53.689 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 07:07:53.688762 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T07:07:53.704 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 07:07:53.704648 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T07:08:03.791 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 07:08:03.791311 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T07:08:03.804 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 07:08:03.804130 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T07:08:13.950 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 07:08:13.950328 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T07:08:13.961 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 07:08:13.961153 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T07:08:24.041 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 07:08:24.040863 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T07:08:24.115 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 07:08:24.114671 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T07:08:34.222 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 07:08:34.219108 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T07:08:34.231 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 07:08:34.231688 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T07:08:44.340 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 07:08:44.340408 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T07:08:44.353 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 07:08:44.353469 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T07:08:54.478 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 07:08:54.477595 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T07:08:54.492 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 07:08:54.492140 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T07:09:04.566 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 07:09:04.565929 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T07:09:04.579 INFO:tasks.rook.operator.smithi029.stdout:2022-04-02 07:09:04.578981 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-02T09:29:55.845 DEBUG:teuthology.exit:Got signal 15; running 1 handler... 2022-04-02T09:29:55.870 DEBUG:teuthology.task.console_log:Killing console logger for smithi029 2022-04-02T09:29:55.871 DEBUG:teuthology.task.console_log:Killing console logger for smithi104 2022-04-02T09:29:55.872 DEBUG:teuthology.task.console_log:Killing console logger for smithi142 2022-04-02T09:29:55.872 DEBUG:teuthology.exit:Finished running handlers