2022-03-10T02:57:41.933 INFO:root:teuthology version: 1.1.0-ff5db916 2022-03-10T02:57:41.936 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_yuriw@teuthology --verbose --archive /home/teuthworker/archive/yuriw-2022-03-10_01:04:51-rados-wip-yuri5-testing-2022-03-07-0958-distro-default-smithi/6728699 --description rados/rook/smoke/{0-distro/ubuntu_20.04 0-kubeadm 0-nvme-loop 1-rook 2-workload/none 3-final cluster/3-node k8s/1.21 net/calico rook/master} --name yuriw-2022-03-10_01:04:51-rados-wip-yuri5-testing-2022-03-07-0958-distro-default-smithi -- /home/teuthworker/archive/yuriw-2022-03-10_01:04:51-rados-wip-yuri5-testing-2022-03-07-0958-distro-default-smithi/6728699/orig.config.yaml 2022-03-10T02:57:41.972 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2022-03-10T02:57:42.037 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2022-03-10_01:04:51-rados-wip-yuri5-testing-2022-03-07-0958-distro-default-smithi/6728699 branch: wip-yuri5-testing-2022-03-07-0958 description: rados/rook/smoke/{0-distro/ubuntu_20.04 0-kubeadm 0-nvme-loop 1-rook 2-workload/none 3-final cluster/3-node k8s/1.21 net/calico rook/master} email: ceph-qa@ceph.io first_in_suite: false job_id: '6728699' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: yuriw-2022-03-10_01:04:51-rados-wip-yuri5-testing-2022-03-07-0958-distro-default-smithi nuke-on-error: true os_type: ubuntu os_version: '20.04' overrides: admin_socket: branch: wip-yuri5-testing-2022-03-07-0958 ceph: conf: mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 7a44e9ebca556862e9704be295a07719d7db47b3 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: flavor: default sha1: 7a44e9ebca556862e9704be295a07719d7db47b3 kernel: hwe: true kubeadm: pod_network: calico version: '1.21' rook: rook_image: rook/ceph:master workunit: branch: wip-yuri5-testing-2022-03-07-0958 sha1: 7a44e9ebca556862e9704be295a07719d7db47b3 owner: scheduled_yuriw@teuthology priority: 71 repo: https://github.com/ceph/ceph-ci.git roles: - - host.a - client.a - - host.b - client.b - - host.c - client.c sha1: 7a44e9ebca556862e9704be295a07719d7db47b3 sleep_before_teardown: 0 suite: rados suite_branch: wip-yuri5-testing-2022-03-07-0958 suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_7a44e9ebca556862e9704be295a07719d7db47b3/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 7a44e9ebca556862e9704be295a07719d7db47b3 targets: smithi037.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCx4zBVFwb3CKrSZnRbA2UD2bNJ34cv5niTPWu0HVYk70gDuIch+V+3Ks1y1v8/5AkwncSNbq/XowGV6CJHP/2rH8UtI9w3bpkijCYZOtaarDi/DBHn2jdI4ExpC9ASxJ03RrfIE5BlpaNBcr1uV/s5Q1DbNni/VodldTkWsv7TYGabSE8rkIoymOKxzGdEdlbQ7+qSsdUe075vhmaygtjrypTllAaj19TyN5Aa/qRAw9xSyrtk2Xxo/kOrAfdfAB8cMnXAhtL+JzdwK+0sGXKOQnP3eGLakhaoDa6gGZyTeZO8ItQUOogj0nt4GZw9q7vE7lyhByRSeTxq12qqkg+7ogM+3iPXqUyDLN9Sco8WKPJ5I7sFQC6U9L2QZplWa6kPSoHHCibdvuLab+GacZrIzUglzPCLxjwFrj5VOpS/qlZ4RM2fu7r/z2DPpJicPT+19xS+Lv8PyzEJUYm1La8Q4X0P27Fv2Rfh37pywCKgOv2ozU5Q8MIj7zewKMo8nTs= smithi038.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCoZdd0OYWdiVwxEgajhI7keS9oQmOICBUcIYkObcT1GurdWAtCOzfFTikRrK8g/vSAgPkfj5iee95WTzsqPvho4qGBPz26zAsB8i8FIFUMZbFcN67/Ja7ebJDCS/DDYROdLfEZhGt72B7KoNJtret48hZ24tjM8gfkJc/KSjKfDeLrVhaEF+jSOvI8RWOvn+UMhXlTL6gurx+Vw508ojCtmUa6korCdl0B5SDHN+KfhLZWPqIPbY6kcJ5XmBgm+/LfMjaKKzIO6RqKH0nhkZEKdd/Tr9dYMBiESmSmbBiC7xywYENg/rbFsFw6o71p0oOmHMnFLJoMfXsuolwJLcvvby9K7DCyPilT7+icFLSs1G00+0B0fktp88jrtaOTZSGi4Icrc6DCr494Pjt3jy8QJ/7vTw6BrXdDutyI11u09CwxPuiO1rHTk08+t4ofBRjvVypfkGtTRjJ7831w+T1ueWLJSjCHxs20PQmb4799T9eRITf/DVbffl/1fgnXJI8= smithi195.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6oSNRRaty7A6pLSUJ/nRMwNWCSqtPH8Ku9CbeYy2ItRIrlLJV0e0y5X7eoSpsg+CAunlWdvwZnBDPZduelxgcCuS5dYYEt9LfLnpZH2IFgYwZp3rK3HLQ8znJdbjM+R2kkHfxH/LJK2tw5YEByUrExfCXEbIwY2k8ha5Kmoilq4M+rk3LaRXvSb7aHba4NMObu3sxmX8geqUoGRTbiW4pOrJWn96HrMO7CPR9tHhYr4WST1NEqe7etvvHWNA/8QgLT8MZ4nZW2E04eqp4ynJZ/B7lOBwsAd+QrSZrFKe83cwK44Oc4/iGce1Z+EFLNtYhNDkp76vZsR9+vRheegDBw+3HNeamgXVzT7QW4v8BktXAzJHxkN6b6qKohRFJQeHWbtlNxxUzWjvArJ7SQHYfUaG+ZZmu4Jrjqrj/GYe+VcPbwMR3RpJetPHwfJxIDyu25rOrCn9P04a32u+jZ17hfz5slZPL1NJbnwdWJryt9DCg35zU4tXDDeyKODXsmrU= tasks: - kubeadm: null - nvme_loop: null - rook: null - rook.shell: - ceph -s - ceph orch status - ceph orch ps - ceph orch ls - ceph orch device ls - exec: host.a: - "set -ex\ntoolbox() {\n kubectl -n rook-ceph exec -it deploy/rook-ceph-tools\ \ -- \"$@\"\n}\norig_num_osd=`toolbox ceph osd stat | cut -f3 -d \" \"`\ntoolbox\ \ ceph orch osd rm 0 --force\nremoved_pv=\"\"\nwhile [ \"$removed_pv\" = \"\"\ \ ]\ndo\n removed_pv=`kubectl get pv | grep Released | cut -f1 -d \" \"`\n\ \ sleep 3s\ndone\ntarget_path=`kubectl get pv $removed_pv -o jsonpath='{.spec.local.path}'`\n\ host=`echo $removed_pv | cut -f1 -d \"-\"`\ntoolbox ceph orch device zap $host\ \ $target_path --force\nzap_completion=\"0\"\nwhile [ \"$zap_completion\" =\ \ \"0\" ]\ndo\n zap_completion=`kubectl get job -n rook-ceph rook-ceph-device-zap\ \ -o jsonpath='{.status.succeeded.path}'`\n sleep 3s\ndone\nkubectl patch\ \ pv $removed_pv -p '{\"spec\":{\"claimRef\": null}}'\ntoolbox ceph orch apply\ \ osd --all-available-devices\nkubectl delete job rook-ceph-device-zap -n rook-ceph\n\ num_osd=\"0\"\nwhile [ \"$num_osd\" != \"$orig_num_osd\" ]\ndo\n echo \"\ waiting for osd to come back up\"\n num_osd=`toolbox ceph osd stat | cut\ \ -f3 -d \" \"`\n sleep 30s\ndone\n" - rook.shell: commands: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - ceph orch apply rgw foo - ceph orch apply mds foo - ceph orch apply rbd-mirror - ceph orch apply nfs foo --port 12777 teuthology_branch: master teuthology_sha1: ff5db9169566776b2f41faf671316ca58a318ea6 timestamp: 2022-03-10_01:04:51 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.22301 2022-03-10T02:57:42.038 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_7a44e9ebca556862e9704be295a07719d7db47b3/qa; will attempt to use it 2022-03-10T02:57:42.038 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_7a44e9ebca556862e9704be295a07719d7db47b3/qa/tasks 2022-03-10T02:57:42.039 INFO:teuthology.run_tasks:Running task internal.check_packages... 2022-03-10T02:57:42.043 INFO:teuthology.task.internal:Checking packages... 2022-03-10T02:57:42.066 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '7a44e9ebca556862e9704be295a07719d7db47b3' 2022-03-10T02:57:42.067 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2022-03-10T02:57:42.067 INFO:teuthology.packaging:ref: None 2022-03-10T02:57:42.067 INFO:teuthology.packaging:tag: None 2022-03-10T02:57:42.068 INFO:teuthology.packaging:branch: wip-yuri5-testing-2022-03-07-0958 2022-03-10T02:57:42.068 INFO:teuthology.packaging:sha1: 7a44e9ebca556862e9704be295a07719d7db47b3 2022-03-10T02:57:42.069 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&ref=wip-yuri5-testing-2022-03-07-0958 2022-03-10T02:57:42.228 INFO:teuthology.task.internal:Found packages for ceph version 17.0.0-11025-g7a44e9eb-1focal 2022-03-10T02:57:42.228 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2022-03-10T02:57:42.259 INFO:teuthology.task.internal:no buildpackages task found 2022-03-10T02:57:42.259 INFO:teuthology.run_tasks:Running task internal.save_config... 2022-03-10T02:57:42.309 INFO:teuthology.task.internal:Saving configuration 2022-03-10T02:57:42.323 INFO:teuthology.run_tasks:Running task internal.check_lock... 2022-03-10T02:57:42.361 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-03-10T02:57:42.383 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi037.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2022-03-10_01:04:51-rados-wip-yuri5-testing-2022-03-07-0958-distro-default-smithi/6728699', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2022-03-10 02:52:30.500434', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIPDRo4wp6mNXqFdCofjVmBcEViHtzw5xDLDi491aRefU'} 2022-03-10T02:57:42.407 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi038.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2022-03-10_01:04:51-rados-wip-yuri5-testing-2022-03-07-0958-distro-default-smithi/6728699', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2022-03-10 02:52:30.499106', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIESudMIqGNuvccENZ4fEvahsIU73cQpsRmbejihvV8U4'} 2022-03-10T02:57:42.429 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi195.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2022-03-10_01:04:51-rados-wip-yuri5-testing-2022-03-07-0958-distro-default-smithi/6728699', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2022-03-10 02:52:30.501450', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDfrIuHekwMnI6s1lk9hIFKih75TNptIjDUk0kIJpIWe'} 2022-03-10T02:57:42.429 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2022-03-10T02:57:42.440 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['host.a', 'client.a'] 2022-03-10T02:57:42.441 INFO:teuthology.task.internal:roles: ubuntu@smithi038.front.sepia.ceph.com - ['host.b', 'client.b'] 2022-03-10T02:57:42.441 INFO:teuthology.task.internal:roles: ubuntu@smithi195.front.sepia.ceph.com - ['host.c', 'client.c'] 2022-03-10T02:57:42.441 INFO:teuthology.run_tasks:Running task console_log... 2022-03-10T02:57:42.593 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f7fac2db9d8>, signals=[15]) 2022-03-10T02:57:42.594 INFO:teuthology.run_tasks:Running task internal.connect... 2022-03-10T02:57:42.603 INFO:teuthology.task.internal:Opening connections... 2022-03-10T02:57:42.604 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi037.front.sepia.ceph.com 2022-03-10T02:57:42.605 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi037.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-10T02:57:42.658 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi038.front.sepia.ceph.com 2022-03-10T02:57:42.659 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi038.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-10T02:57:42.714 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi195.front.sepia.ceph.com 2022-03-10T02:57:42.715 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi195.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-10T02:57:42.773 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2022-03-10T02:57:42.784 DEBUG:teuthology.orchestra.run.smithi037:> uname -m 2022-03-10T02:57:42.792 INFO:teuthology.orchestra.run.smithi037.stdout:x86_64 2022-03-10T02:57:42.793 DEBUG:teuthology.orchestra.run.smithi037:> cat /etc/os-release 2022-03-10T02:57:42.842 INFO:teuthology.orchestra.run.smithi037.stdout:NAME="Ubuntu" 2022-03-10T02:57:42.842 INFO:teuthology.orchestra.run.smithi037.stdout:VERSION="20.04.3 LTS (Focal Fossa)" 2022-03-10T02:57:42.843 INFO:teuthology.orchestra.run.smithi037.stdout:ID=ubuntu 2022-03-10T02:57:42.843 INFO:teuthology.orchestra.run.smithi037.stdout:ID_LIKE=debian 2022-03-10T02:57:42.843 INFO:teuthology.orchestra.run.smithi037.stdout:PRETTY_NAME="Ubuntu 20.04.3 LTS" 2022-03-10T02:57:42.843 INFO:teuthology.orchestra.run.smithi037.stdout:VERSION_ID="20.04" 2022-03-10T02:57:42.844 INFO:teuthology.orchestra.run.smithi037.stdout:HOME_URL="https://www.ubuntu.com/" 2022-03-10T02:57:42.844 INFO:teuthology.orchestra.run.smithi037.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2022-03-10T02:57:42.844 INFO:teuthology.orchestra.run.smithi037.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2022-03-10T02:57:42.845 INFO:teuthology.orchestra.run.smithi037.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2022-03-10T02:57:42.845 INFO:teuthology.orchestra.run.smithi037.stdout:VERSION_CODENAME=focal 2022-03-10T02:57:42.845 INFO:teuthology.orchestra.run.smithi037.stdout:UBUNTU_CODENAME=focal 2022-03-10T02:57:42.846 INFO:teuthology.lock.ops:Updating smithi037.front.sepia.ceph.com on lock server 2022-03-10T02:57:42.864 DEBUG:teuthology.orchestra.run.smithi038:> uname -m 2022-03-10T02:57:42.871 INFO:teuthology.orchestra.run.smithi038.stdout:x86_64 2022-03-10T02:57:42.872 DEBUG:teuthology.orchestra.run.smithi038:> cat /etc/os-release 2022-03-10T02:57:42.921 INFO:teuthology.orchestra.run.smithi038.stdout:NAME="Ubuntu" 2022-03-10T02:57:42.921 INFO:teuthology.orchestra.run.smithi038.stdout:VERSION="20.04.3 LTS (Focal Fossa)" 2022-03-10T02:57:42.921 INFO:teuthology.orchestra.run.smithi038.stdout:ID=ubuntu 2022-03-10T02:57:42.922 INFO:teuthology.orchestra.run.smithi038.stdout:ID_LIKE=debian 2022-03-10T02:57:42.922 INFO:teuthology.orchestra.run.smithi038.stdout:PRETTY_NAME="Ubuntu 20.04.3 LTS" 2022-03-10T02:57:42.922 INFO:teuthology.orchestra.run.smithi038.stdout:VERSION_ID="20.04" 2022-03-10T02:57:42.922 INFO:teuthology.orchestra.run.smithi038.stdout:HOME_URL="https://www.ubuntu.com/" 2022-03-10T02:57:42.922 INFO:teuthology.orchestra.run.smithi038.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2022-03-10T02:57:42.923 INFO:teuthology.orchestra.run.smithi038.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2022-03-10T02:57:42.923 INFO:teuthology.orchestra.run.smithi038.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2022-03-10T02:57:42.923 INFO:teuthology.orchestra.run.smithi038.stdout:VERSION_CODENAME=focal 2022-03-10T02:57:42.924 INFO:teuthology.orchestra.run.smithi038.stdout:UBUNTU_CODENAME=focal 2022-03-10T02:57:42.924 INFO:teuthology.lock.ops:Updating smithi038.front.sepia.ceph.com on lock server 2022-03-10T02:57:42.943 DEBUG:teuthology.orchestra.run.smithi195:> uname -m 2022-03-10T02:57:42.950 INFO:teuthology.orchestra.run.smithi195.stdout:x86_64 2022-03-10T02:57:42.951 DEBUG:teuthology.orchestra.run.smithi195:> cat /etc/os-release 2022-03-10T02:57:42.997 INFO:teuthology.orchestra.run.smithi195.stdout:NAME="Ubuntu" 2022-03-10T02:57:42.997 INFO:teuthology.orchestra.run.smithi195.stdout:VERSION="20.04.3 LTS (Focal Fossa)" 2022-03-10T02:57:42.998 INFO:teuthology.orchestra.run.smithi195.stdout:ID=ubuntu 2022-03-10T02:57:42.998 INFO:teuthology.orchestra.run.smithi195.stdout:ID_LIKE=debian 2022-03-10T02:57:42.998 INFO:teuthology.orchestra.run.smithi195.stdout:PRETTY_NAME="Ubuntu 20.04.3 LTS" 2022-03-10T02:57:42.998 INFO:teuthology.orchestra.run.smithi195.stdout:VERSION_ID="20.04" 2022-03-10T02:57:42.999 INFO:teuthology.orchestra.run.smithi195.stdout:HOME_URL="https://www.ubuntu.com/" 2022-03-10T02:57:42.999 INFO:teuthology.orchestra.run.smithi195.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2022-03-10T02:57:42.999 INFO:teuthology.orchestra.run.smithi195.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2022-03-10T02:57:42.999 INFO:teuthology.orchestra.run.smithi195.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2022-03-10T02:57:42.999 INFO:teuthology.orchestra.run.smithi195.stdout:VERSION_CODENAME=focal 2022-03-10T02:57:43.000 INFO:teuthology.orchestra.run.smithi195.stdout:UBUNTU_CODENAME=focal 2022-03-10T02:57:43.000 INFO:teuthology.lock.ops:Updating smithi195.front.sepia.ceph.com on lock server 2022-03-10T02:57:43.018 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2022-03-10T02:57:43.032 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2022-03-10T02:57:43.072 INFO:teuthology.task.internal:Checking for old test directory... 2022-03-10T02:57:43.073 DEBUG:teuthology.orchestra.run.smithi037:> test '!' -e /home/ubuntu/cephtest 2022-03-10T02:57:43.076 DEBUG:teuthology.orchestra.run.smithi038:> test '!' -e /home/ubuntu/cephtest 2022-03-10T02:57:43.078 DEBUG:teuthology.orchestra.run.smithi195:> test '!' -e /home/ubuntu/cephtest 2022-03-10T02:57:43.085 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2022-03-10T02:57:43.106 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2022-03-10T02:57:43.106 DEBUG:teuthology.orchestra.run.smithi037:> test -z $(ls -A /var/lib/ceph) 2022-03-10T02:57:43.126 DEBUG:teuthology.orchestra.run.smithi038:> test -z $(ls -A /var/lib/ceph) 2022-03-10T02:57:43.128 DEBUG:teuthology.orchestra.run.smithi195:> test -z $(ls -A /var/lib/ceph) 2022-03-10T02:57:43.145 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2022-03-10T02:57:43.225 INFO:teuthology.run_tasks:Running task kernel... 2022-03-10T02:57:43.243 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2022-03-10T02:57:43.244 INFO:teuthology.task.kernel:normalize config orig: {'hwe': True} 2022-03-10T02:57:43.245 DEBUG:teuthology.task.kernel:normalized overrides {'host.a': {'hwe': True}, 'client.a': {'hwe': True}, 'host.b': {'hwe': True}, 'client.b': {'hwe': True}, 'host.c': {'hwe': True}, 'client.c': {'hwe': True}} 2022-03-10T02:57:43.245 INFO:teuthology.task.kernel:config {'host.a': {'kdb': True, 'sha1': 'distro', 'hwe': True}, 'host.b': {'kdb': True, 'sha1': 'distro', 'hwe': True}, 'host.c': {'kdb': True, 'sha1': 'distro', 'hwe': True}}, timeout 300 2022-03-10T02:57:43.245 DEBUG:teuthology.orchestra.run.smithi037:> uname -r 2022-03-10T02:57:43.246 DEBUG:teuthology.orchestra.run.smithi038:> uname -r 2022-03-10T02:57:43.247 DEBUG:teuthology.orchestra.run.smithi195:> uname -r 2022-03-10T02:57:43.254 INFO:teuthology.orchestra.run.smithi038.stdout:5.4.0-97-generic 2022-03-10T02:57:43.254 INFO:teuthology.task.kernel:Running kernel on smithi038: 5.4.0-97-generic 2022-03-10T02:57:43.255 DEBUG:teuthology.orchestra.run.smithi038:> sudo apt-get clean 2022-03-10T02:57:43.257 INFO:teuthology.orchestra.run.smithi195.stdout:5.4.0-97-generic 2022-03-10T02:57:43.257 INFO:teuthology.orchestra.run.smithi037.stdout:5.4.0-97-generic 2022-03-10T02:57:43.257 INFO:teuthology.task.kernel:Running kernel on smithi195: 5.4.0-97-generic 2022-03-10T02:57:43.258 DEBUG:teuthology.orchestra.run.smithi195:> sudo apt-get clean 2022-03-10T02:57:43.258 INFO:teuthology.task.kernel:Running kernel on smithi037: 5.4.0-97-generic 2022-03-10T02:57:43.258 DEBUG:teuthology.orchestra.run.smithi037:> sudo apt-get clean 2022-03-10T02:57:43.369 DEBUG:teuthology.orchestra.run.smithi038:> sudo apt-get update 2022-03-10T02:57:43.380 DEBUG:teuthology.orchestra.run.smithi195:> sudo apt-get update 2022-03-10T02:57:43.416 DEBUG:teuthology.orchestra.run.smithi037:> sudo apt-get update 2022-03-10T02:57:43.668 INFO:teuthology.orchestra.run.smithi195.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-10T02:57:43.668 INFO:teuthology.orchestra.run.smithi195.stdout:Get:2 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2022-03-10T02:57:43.692 INFO:teuthology.orchestra.run.smithi038.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-10T02:57:43.692 INFO:teuthology.orchestra.run.smithi038.stdout:Get:2 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2022-03-10T02:57:43.723 INFO:teuthology.orchestra.run.smithi037.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-10T02:57:43.724 INFO:teuthology.orchestra.run.smithi037.stdout:Get:2 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2022-03-10T02:57:43.728 INFO:teuthology.orchestra.run.smithi038.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2022-03-10T02:57:43.728 INFO:teuthology.orchestra.run.smithi195.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2022-03-10T02:57:43.778 INFO:teuthology.orchestra.run.smithi037.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2022-03-10T02:57:44.075 INFO:teuthology.orchestra.run.smithi038.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2022-03-10T02:57:44.102 INFO:teuthology.orchestra.run.smithi195.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2022-03-10T02:57:44.121 INFO:teuthology.orchestra.run.smithi037.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2022-03-10T02:57:44.150 INFO:teuthology.orchestra.run.smithi195.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [395 kB] 2022-03-10T02:57:44.158 INFO:teuthology.orchestra.run.smithi038.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [395 kB] 2022-03-10T02:57:44.195 INFO:teuthology.orchestra.run.smithi037.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [1,310 kB] 2022-03-10T02:57:44.294 INFO:teuthology.orchestra.run.smithi195.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [1,640 kB] 2022-03-10T02:57:44.301 INFO:teuthology.orchestra.run.smithi038.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [1,640 kB] 2022-03-10T02:57:44.347 INFO:teuthology.orchestra.run.smithi037.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [615 kB] 2022-03-10T02:57:44.423 INFO:teuthology.orchestra.run.smithi195.stdout:Get:7 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [1,310 kB] 2022-03-10T02:57:44.437 INFO:teuthology.orchestra.run.smithi038.stdout:Get:7 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [1,310 kB] 2022-03-10T02:57:44.575 INFO:teuthology.orchestra.run.smithi195.stdout:Get:8 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [228 kB] 2022-03-10T02:57:44.591 INFO:teuthology.orchestra.run.smithi038.stdout:Get:8 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [228 kB] 2022-03-10T02:57:44.593 INFO:teuthology.orchestra.run.smithi037.stdout:Get:7 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [395 kB] 2022-03-10T02:57:44.595 INFO:teuthology.orchestra.run.smithi195.stdout:Get:9 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [9,732 B] 2022-03-10T02:57:44.596 INFO:teuthology.orchestra.run.smithi195.stdout:Get:10 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [799 kB] 2022-03-10T02:57:44.610 INFO:teuthology.orchestra.run.smithi037.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [1,640 kB] 2022-03-10T02:57:44.611 INFO:teuthology.orchestra.run.smithi037.stdout:Get:9 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [228 kB] 2022-03-10T02:57:44.612 INFO:teuthology.orchestra.run.smithi038.stdout:Get:9 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [9,732 B] 2022-03-10T02:57:44.612 INFO:teuthology.orchestra.run.smithi038.stdout:Get:10 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [21.7 kB] 2022-03-10T02:57:44.613 INFO:teuthology.orchestra.run.smithi038.stdout:Get:11 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [799 kB] 2022-03-10T02:57:44.624 INFO:teuthology.orchestra.run.smithi037.stdout:Get:10 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [9,732 B] 2022-03-10T02:57:44.624 INFO:teuthology.orchestra.run.smithi037.stdout:Get:11 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [799 kB] 2022-03-10T02:57:44.641 INFO:teuthology.orchestra.run.smithi195.stdout:Get:11 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [21.7 kB] 2022-03-10T02:57:44.642 INFO:teuthology.orchestra.run.smithi195.stdout:Get:12 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [114 kB] 2022-03-10T02:57:44.647 INFO:teuthology.orchestra.run.smithi195.stdout:Get:13 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [504 B] 2022-03-10T02:57:44.647 INFO:teuthology.orchestra.run.smithi195.stdout:Get:14 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [536 kB] 2022-03-10T02:57:44.654 INFO:teuthology.orchestra.run.smithi195.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [615 kB] 2022-03-10T02:57:44.661 INFO:teuthology.orchestra.run.smithi038.stdout:Get:12 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [114 kB] 2022-03-10T02:57:44.666 INFO:teuthology.orchestra.run.smithi038.stdout:Get:13 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [504 B] 2022-03-10T02:57:44.666 INFO:teuthology.orchestra.run.smithi038.stdout:Get:14 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [536 kB] 2022-03-10T02:57:44.668 INFO:teuthology.orchestra.run.smithi195.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [312 kB] 2022-03-10T02:57:44.669 INFO:teuthology.orchestra.run.smithi195.stdout:Get:17 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [681 kB] 2022-03-10T02:57:44.673 INFO:teuthology.orchestra.run.smithi195.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [14.8 kB] 2022-03-10T02:57:44.674 INFO:teuthology.orchestra.run.smithi195.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [853 kB] 2022-03-10T02:57:44.676 INFO:teuthology.orchestra.run.smithi038.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [615 kB] 2022-03-10T02:57:44.691 INFO:teuthology.orchestra.run.smithi038.stdout:Get:16 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [681 kB] 2022-03-10T02:57:44.697 INFO:teuthology.orchestra.run.smithi037.stdout:Get:12 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [21.7 kB] 2022-03-10T02:57:44.698 INFO:teuthology.orchestra.run.smithi195.stdout:Get:20 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [117 kB] 2022-03-10T02:57:44.698 INFO:teuthology.orchestra.run.smithi038.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [312 kB] 2022-03-10T02:57:44.699 INFO:teuthology.orchestra.run.smithi037.stdout:Get:13 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [114 kB] 2022-03-10T02:57:44.701 INFO:teuthology.orchestra.run.smithi195.stdout:Get:21 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [13.2 kB] 2022-03-10T02:57:44.704 INFO:teuthology.orchestra.run.smithi037.stdout:Get:14 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [504 B] 2022-03-10T02:57:44.704 INFO:teuthology.orchestra.run.smithi037.stdout:Get:15 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [536 kB] 2022-03-10T02:57:44.705 INFO:teuthology.orchestra.run.smithi038.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [14.8 kB] 2022-03-10T02:57:44.706 INFO:teuthology.orchestra.run.smithi038.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [853 kB] 2022-03-10T02:57:44.712 INFO:teuthology.orchestra.run.smithi038.stdout:Get:20 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [117 kB] 2022-03-10T02:57:44.714 INFO:teuthology.orchestra.run.smithi038.stdout:Get:21 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [13.2 kB] 2022-03-10T02:57:44.719 INFO:teuthology.orchestra.run.smithi037.stdout:Get:16 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [681 kB] 2022-03-10T02:57:44.736 INFO:teuthology.orchestra.run.smithi195.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [23.1 kB] 2022-03-10T02:57:44.737 INFO:teuthology.orchestra.run.smithi195.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [122 kB] 2022-03-10T02:57:44.739 INFO:teuthology.orchestra.run.smithi195.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [500 B] 2022-03-10T02:57:44.739 INFO:teuthology.orchestra.run.smithi195.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [671 kB] 2022-03-10T02:57:44.742 INFO:teuthology.orchestra.run.smithi037.stdout:Get:17 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [117 kB] 2022-03-10T02:57:44.745 INFO:teuthology.orchestra.run.smithi195.stdout:Get:26 http://security.ubuntu.com/ubuntu focal-security/multiverse i386 Packages [7,180 B] 2022-03-10T02:57:44.745 INFO:teuthology.orchestra.run.smithi195.stdout:Get:27 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [20.7 kB] 2022-03-10T02:57:44.746 INFO:teuthology.orchestra.run.smithi037.stdout:Get:18 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [13.2 kB] 2022-03-10T02:57:44.746 INFO:teuthology.orchestra.run.smithi195.stdout:Get:28 http://security.ubuntu.com/ubuntu focal-security/multiverse Translation-en [5,196 B] 2022-03-10T02:57:44.747 INFO:teuthology.orchestra.run.smithi195.stdout:Get:29 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 c-n-f Metadata [500 B] 2022-03-10T02:57:44.751 INFO:teuthology.orchestra.run.smithi195.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [908 kB] 2022-03-10T02:57:44.758 INFO:teuthology.orchestra.run.smithi038.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [23.1 kB] 2022-03-10T02:57:44.759 INFO:teuthology.orchestra.run.smithi038.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [122 kB] 2022-03-10T02:57:44.761 INFO:teuthology.orchestra.run.smithi038.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [500 B] 2022-03-10T02:57:44.762 INFO:teuthology.orchestra.run.smithi038.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [908 kB] 2022-03-10T02:57:44.762 INFO:teuthology.orchestra.run.smithi038.stdout:Get:26 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [20.7 kB] 2022-03-10T02:57:44.763 INFO:teuthology.orchestra.run.smithi038.stdout:Get:27 http://security.ubuntu.com/ubuntu focal-security/multiverse i386 Packages [7,180 B] 2022-03-10T02:57:44.763 INFO:teuthology.orchestra.run.smithi038.stdout:Get:28 http://security.ubuntu.com/ubuntu focal-security/multiverse Translation-en [5,196 B] 2022-03-10T02:57:44.764 INFO:teuthology.orchestra.run.smithi038.stdout:Get:29 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 c-n-f Metadata [500 B] 2022-03-10T02:57:44.772 INFO:teuthology.orchestra.run.smithi195.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [202 kB] 2022-03-10T02:57:44.773 INFO:teuthology.orchestra.run.smithi037.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [312 kB] 2022-03-10T02:57:44.777 INFO:teuthology.orchestra.run.smithi195.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [20.2 kB] 2022-03-10T02:57:44.779 INFO:teuthology.orchestra.run.smithi037.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [14.8 kB] 2022-03-10T02:57:44.780 INFO:teuthology.orchestra.run.smithi037.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [23.1 kB] 2022-03-10T02:57:44.780 INFO:teuthology.orchestra.run.smithi038.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [671 kB] 2022-03-10T02:57:44.781 INFO:teuthology.orchestra.run.smithi037.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [853 kB] 2022-03-10T02:57:44.788 INFO:teuthology.orchestra.run.smithi037.stdout:Get:23 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [20.7 kB] 2022-03-10T02:57:44.789 INFO:teuthology.orchestra.run.smithi037.stdout:Get:24 http://security.ubuntu.com/ubuntu focal-security/multiverse i386 Packages [7,180 B] 2022-03-10T02:57:44.790 INFO:teuthology.orchestra.run.smithi037.stdout:Get:25 http://security.ubuntu.com/ubuntu focal-security/multiverse Translation-en [5,196 B] 2022-03-10T02:57:44.790 INFO:teuthology.orchestra.run.smithi037.stdout:Get:26 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 c-n-f Metadata [500 B] 2022-03-10T02:57:44.793 INFO:teuthology.orchestra.run.smithi038.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [202 kB] 2022-03-10T02:57:44.798 INFO:teuthology.orchestra.run.smithi038.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [20.2 kB] 2022-03-10T02:57:44.810 INFO:teuthology.orchestra.run.smithi037.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [122 kB] 2022-03-10T02:57:44.813 INFO:teuthology.orchestra.run.smithi037.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [500 B] 2022-03-10T02:57:44.813 INFO:teuthology.orchestra.run.smithi037.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [671 kB] 2022-03-10T02:57:44.822 INFO:teuthology.orchestra.run.smithi195.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [23.8 kB] 2022-03-10T02:57:44.823 INFO:teuthology.orchestra.run.smithi195.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8,460 B] 2022-03-10T02:57:44.826 INFO:teuthology.orchestra.run.smithi195.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [7,312 B] 2022-03-10T02:57:44.830 INFO:teuthology.orchestra.run.smithi195.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [580 B] 2022-03-10T02:57:44.832 INFO:teuthology.orchestra.run.smithi195.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [42.2 kB] 2022-03-10T02:57:44.835 INFO:teuthology.orchestra.run.smithi037.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [908 kB] 2022-03-10T02:57:44.837 INFO:teuthology.orchestra.run.smithi195.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal-backports/main i386 Packages [34.8 kB] 2022-03-10T02:57:44.844 INFO:teuthology.orchestra.run.smithi038.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [23.8 kB] 2022-03-10T02:57:44.845 INFO:teuthology.orchestra.run.smithi038.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8,460 B] 2022-03-10T02:57:44.846 INFO:teuthology.orchestra.run.smithi038.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [7,312 B] 2022-03-10T02:57:44.846 INFO:teuthology.orchestra.run.smithi038.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [580 B] 2022-03-10T02:57:44.847 INFO:teuthology.orchestra.run.smithi038.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [42.2 kB] 2022-03-10T02:57:44.862 INFO:teuthology.orchestra.run.smithi038.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal-backports/main i386 Packages [34.8 kB] 2022-03-10T02:57:44.867 INFO:teuthology.orchestra.run.smithi037.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [202 kB] 2022-03-10T02:57:44.871 INFO:teuthology.orchestra.run.smithi037.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [20.2 kB] 2022-03-10T02:57:44.896 INFO:teuthology.orchestra.run.smithi037.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8,460 B] 2022-03-10T02:57:44.897 INFO:teuthology.orchestra.run.smithi037.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [23.8 kB] 2022-03-10T02:57:44.897 INFO:teuthology.orchestra.run.smithi037.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [7,312 B] 2022-03-10T02:57:44.898 INFO:teuthology.orchestra.run.smithi037.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [580 B] 2022-03-10T02:57:44.898 INFO:teuthology.orchestra.run.smithi037.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [42.2 kB] 2022-03-10T02:57:44.915 INFO:teuthology.orchestra.run.smithi195.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal-backports/main Translation-en [10.1 kB] 2022-03-10T02:57:44.915 INFO:teuthology.orchestra.run.smithi195.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [22.7 kB] 2022-03-10T02:57:44.916 INFO:teuthology.orchestra.run.smithi195.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [12.9 kB] 2022-03-10T02:57:44.917 INFO:teuthology.orchestra.run.smithi195.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal-backports/universe Translation-en [15.4 kB] 2022-03-10T02:57:44.918 INFO:teuthology.orchestra.run.smithi195.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [804 B] 2022-03-10T02:57:44.939 INFO:teuthology.orchestra.run.smithi037.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal-backports/main i386 Packages [34.8 kB] 2022-03-10T02:57:44.939 INFO:teuthology.orchestra.run.smithi037.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal-backports/main Translation-en [10.1 kB] 2022-03-10T02:57:44.944 INFO:teuthology.orchestra.run.smithi038.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal-backports/main Translation-en [10.1 kB] 2022-03-10T02:57:44.945 INFO:teuthology.orchestra.run.smithi038.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [12.9 kB] 2022-03-10T02:57:44.945 INFO:teuthology.orchestra.run.smithi038.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [22.7 kB] 2022-03-10T02:57:44.946 INFO:teuthology.orchestra.run.smithi038.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal-backports/universe Translation-en [15.4 kB] 2022-03-10T02:57:44.947 INFO:teuthology.orchestra.run.smithi038.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [804 B] 2022-03-10T02:57:45.020 INFO:teuthology.orchestra.run.smithi037.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [12.9 kB] 2022-03-10T02:57:45.023 INFO:teuthology.orchestra.run.smithi037.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [22.7 kB] 2022-03-10T02:57:45.023 INFO:teuthology.orchestra.run.smithi037.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal-backports/universe Translation-en [15.4 kB] 2022-03-10T02:57:45.023 INFO:teuthology.orchestra.run.smithi037.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [804 B] 2022-03-10T02:57:47.654 INFO:teuthology.orchestra.run.smithi195.stdout:Fetched 10.2 MB in 2s (4,635 kB/s) 2022-03-10T02:57:47.780 INFO:teuthology.orchestra.run.smithi037.stdout:Fetched 10.2 MB in 2s (4,703 kB/s) 2022-03-10T02:57:47.963 INFO:teuthology.orchestra.run.smithi038.stdout:Fetched 10.2 MB in 2s (4,503 kB/s) 2022-03-10T02:57:48.789 INFO:teuthology.orchestra.run.smithi195.stdout:Reading package lists... 2022-03-10T02:57:48.813 DEBUG:teuthology.orchestra.run.smithi195:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic-hwe-20.04 2022-03-10T02:57:48.879 INFO:teuthology.orchestra.run.smithi195.stdout:Reading package lists... 2022-03-10T02:57:48.992 INFO:teuthology.orchestra.run.smithi037.stdout:Reading package lists... 2022-03-10T02:57:49.016 DEBUG:teuthology.orchestra.run.smithi037:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic-hwe-20.04 2022-03-10T02:57:49.050 INFO:teuthology.orchestra.run.smithi195.stdout:Building dependency tree... 2022-03-10T02:57:49.051 INFO:teuthology.orchestra.run.smithi195.stdout:Reading state information... 2022-03-10T02:57:49.085 INFO:teuthology.orchestra.run.smithi037.stdout:Reading package lists... 2022-03-10T02:57:49.195 INFO:teuthology.orchestra.run.smithi195.stdout:The following additional packages will be installed: 2022-03-10T02:57:49.195 INFO:teuthology.orchestra.run.smithi195.stdout: libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 2022-03-10T02:57:49.196 INFO:teuthology.orchestra.run.smithi195.stdout: libusbmuxd6 linux-image-5.13.0-35-generic linux-modules-5.13.0-35-generic 2022-03-10T02:57:49.196 INFO:teuthology.orchestra.run.smithi195.stdout: linux-modules-extra-5.13.0-35-generic thermald upower usbmuxd 2022-03-10T02:57:49.197 INFO:teuthology.orchestra.run.smithi195.stdout:Suggested packages: 2022-03-10T02:57:49.197 INFO:teuthology.orchestra.run.smithi195.stdout: libusbmuxd-tools fdutils linux-doc | linux-hwe-5.13-source-5.13.0 2022-03-10T02:57:49.197 INFO:teuthology.orchestra.run.smithi195.stdout: linux-hwe-5.13-tools linux-headers-5.13.0-35-generic 2022-03-10T02:57:49.216 INFO:teuthology.orchestra.run.smithi038.stdout:Reading package lists... 2022-03-10T02:57:49.239 DEBUG:teuthology.orchestra.run.smithi038:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic-hwe-20.04 2022-03-10T02:57:49.243 INFO:teuthology.orchestra.run.smithi195.stdout:The following NEW packages will be installed: 2022-03-10T02:57:49.244 INFO:teuthology.orchestra.run.smithi195.stdout: libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 2022-03-10T02:57:49.244 INFO:teuthology.orchestra.run.smithi195.stdout: libusbmuxd6 linux-image-5.13.0-35-generic linux-image-generic-hwe-20.04 2022-03-10T02:57:49.244 INFO:teuthology.orchestra.run.smithi195.stdout: linux-modules-5.13.0-35-generic linux-modules-extra-5.13.0-35-generic 2022-03-10T02:57:49.245 INFO:teuthology.orchestra.run.smithi195.stdout: thermald upower usbmuxd 2022-03-10T02:57:49.265 INFO:teuthology.orchestra.run.smithi037.stdout:Building dependency tree... 2022-03-10T02:57:49.265 INFO:teuthology.orchestra.run.smithi037.stdout:Reading state information... 2022-03-10T02:57:49.308 INFO:teuthology.orchestra.run.smithi038.stdout:Reading package lists... 2022-03-10T02:57:49.414 INFO:teuthology.orchestra.run.smithi037.stdout:The following additional packages will be installed: 2022-03-10T02:57:49.414 INFO:teuthology.orchestra.run.smithi037.stdout: libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 2022-03-10T02:57:49.414 INFO:teuthology.orchestra.run.smithi037.stdout: libusbmuxd6 linux-image-5.13.0-35-generic linux-modules-5.13.0-35-generic 2022-03-10T02:57:49.415 INFO:teuthology.orchestra.run.smithi037.stdout: linux-modules-extra-5.13.0-35-generic thermald upower usbmuxd 2022-03-10T02:57:49.415 INFO:teuthology.orchestra.run.smithi037.stdout:Suggested packages: 2022-03-10T02:57:49.415 INFO:teuthology.orchestra.run.smithi037.stdout: libusbmuxd-tools fdutils linux-doc | linux-hwe-5.13-source-5.13.0 2022-03-10T02:57:49.415 INFO:teuthology.orchestra.run.smithi037.stdout: linux-hwe-5.13-tools linux-headers-5.13.0-35-generic 2022-03-10T02:57:49.427 INFO:teuthology.orchestra.run.smithi195.stdout:0 upgraded, 13 newly installed, 0 to remove and 94 not upgraded. 2022-03-10T02:57:49.427 INFO:teuthology.orchestra.run.smithi195.stdout:Need to get 80.1 MB of archives. 2022-03-10T02:57:49.427 INFO:teuthology.orchestra.run.smithi195.stdout:After this operation, 409 MB of additional disk space will be used. 2022-03-10T02:57:49.427 INFO:teuthology.orchestra.run.smithi195.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 libdbus-glib-1-2 amd64 0.110-5fakssync1 [59.1 kB] 2022-03-10T02:57:49.465 INFO:teuthology.orchestra.run.smithi037.stdout:The following NEW packages will be installed: 2022-03-10T02:57:49.466 INFO:teuthology.orchestra.run.smithi037.stdout: libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 2022-03-10T02:57:49.466 INFO:teuthology.orchestra.run.smithi037.stdout: libusbmuxd6 linux-image-5.13.0-35-generic linux-image-generic-hwe-20.04 2022-03-10T02:57:49.466 INFO:teuthology.orchestra.run.smithi037.stdout: linux-modules-5.13.0-35-generic linux-modules-extra-5.13.0-35-generic 2022-03-10T02:57:49.467 INFO:teuthology.orchestra.run.smithi037.stdout: thermald upower usbmuxd 2022-03-10T02:57:49.484 INFO:teuthology.orchestra.run.smithi038.stdout:Building dependency tree... 2022-03-10T02:57:49.484 INFO:teuthology.orchestra.run.smithi038.stdout:Reading state information... 2022-03-10T02:57:49.627 INFO:teuthology.orchestra.run.smithi038.stdout:The following additional packages will be installed: 2022-03-10T02:57:49.627 INFO:teuthology.orchestra.run.smithi038.stdout: libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 2022-03-10T02:57:49.627 INFO:teuthology.orchestra.run.smithi038.stdout: libusbmuxd6 linux-image-5.13.0-35-generic linux-modules-5.13.0-35-generic 2022-03-10T02:57:49.627 INFO:teuthology.orchestra.run.smithi038.stdout: linux-modules-extra-5.13.0-35-generic thermald upower usbmuxd 2022-03-10T02:57:49.628 INFO:teuthology.orchestra.run.smithi038.stdout:Suggested packages: 2022-03-10T02:57:49.628 INFO:teuthology.orchestra.run.smithi038.stdout: libusbmuxd-tools fdutils linux-doc | linux-hwe-5.13-source-5.13.0 2022-03-10T02:57:49.628 INFO:teuthology.orchestra.run.smithi038.stdout: linux-hwe-5.13-tools linux-headers-5.13.0-35-generic 2022-03-10T02:57:49.673 INFO:teuthology.orchestra.run.smithi038.stdout:The following NEW packages will be installed: 2022-03-10T02:57:49.675 INFO:teuthology.orchestra.run.smithi038.stdout: libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 2022-03-10T02:57:49.675 INFO:teuthology.orchestra.run.smithi038.stdout: libusbmuxd6 linux-image-5.13.0-35-generic linux-image-generic-hwe-20.04 2022-03-10T02:57:49.675 INFO:teuthology.orchestra.run.smithi038.stdout: linux-modules-5.13.0-35-generic linux-modules-extra-5.13.0-35-generic 2022-03-10T02:57:49.675 INFO:teuthology.orchestra.run.smithi038.stdout: thermald upower usbmuxd 2022-03-10T02:57:49.695 INFO:teuthology.orchestra.run.smithi195.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 libplist3 amd64 2.1.0-4build2 [31.6 kB] 2022-03-10T02:57:49.699 INFO:teuthology.orchestra.run.smithi195.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 libusbmuxd6 amd64 2.0.1-2 [19.1 kB] 2022-03-10T02:57:49.723 INFO:teuthology.orchestra.run.smithi037.stdout:0 upgraded, 13 newly installed, 0 to remove and 94 not upgraded. 2022-03-10T02:57:49.723 INFO:teuthology.orchestra.run.smithi037.stdout:Need to get 80.1 MB of archives. 2022-03-10T02:57:49.723 INFO:teuthology.orchestra.run.smithi037.stdout:After this operation, 409 MB of additional disk space will be used. 2022-03-10T02:57:49.724 INFO:teuthology.orchestra.run.smithi037.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 libdbus-glib-1-2 amd64 0.110-5fakssync1 [59.1 kB] 2022-03-10T02:57:49.726 INFO:teuthology.orchestra.run.smithi195.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 libimobiledevice6 amd64 1.2.1~git20191129.9f79242-1build1 [65.2 kB] 2022-03-10T02:57:49.809 INFO:teuthology.orchestra.run.smithi195.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 libupower-glib3 amd64 0.99.11-1build2 [43.2 kB] 2022-03-10T02:57:49.839 INFO:teuthology.orchestra.run.smithi195.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.13.0-35-generic amd64 5.13.0-35.40~20.04.1 [18.3 MB] 2022-03-10T02:57:49.857 INFO:teuthology.orchestra.run.smithi038.stdout:0 upgraded, 13 newly installed, 0 to remove and 94 not upgraded. 2022-03-10T02:57:49.857 INFO:teuthology.orchestra.run.smithi038.stdout:Need to get 80.1 MB of archives. 2022-03-10T02:57:49.857 INFO:teuthology.orchestra.run.smithi038.stdout:After this operation, 409 MB of additional disk space will be used. 2022-03-10T02:57:49.858 INFO:teuthology.orchestra.run.smithi038.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 libdbus-glib-1-2 amd64 0.110-5fakssync1 [59.1 kB] 2022-03-10T02:57:50.046 INFO:teuthology.orchestra.run.smithi037.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 libplist3 amd64 2.1.0-4build2 [31.6 kB] 2022-03-10T02:57:50.071 INFO:teuthology.orchestra.run.smithi037.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 libusbmuxd6 amd64 2.0.1-2 [19.1 kB] 2022-03-10T02:57:50.072 INFO:teuthology.orchestra.run.smithi037.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 libimobiledevice6 amd64 1.2.1~git20191129.9f79242-1build1 [65.2 kB] 2022-03-10T02:57:50.126 INFO:teuthology.orchestra.run.smithi038.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 libplist3 amd64 2.1.0-4build2 [31.6 kB] 2022-03-10T02:57:50.127 INFO:teuthology.orchestra.run.smithi038.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 libusbmuxd6 amd64 2.0.1-2 [19.1 kB] 2022-03-10T02:57:50.141 INFO:teuthology.orchestra.run.smithi038.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 libimobiledevice6 amd64 1.2.1~git20191129.9f79242-1build1 [65.2 kB] 2022-03-10T02:57:50.159 INFO:teuthology.orchestra.run.smithi037.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 libupower-glib3 amd64 0.99.11-1build2 [43.2 kB] 2022-03-10T02:57:50.175 INFO:teuthology.orchestra.run.smithi037.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.13.0-35-generic amd64 5.13.0-35.40~20.04.1 [18.3 MB] 2022-03-10T02:57:50.238 INFO:teuthology.orchestra.run.smithi038.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 libupower-glib3 amd64 0.99.11-1build2 [43.2 kB] 2022-03-10T02:57:50.239 INFO:teuthology.orchestra.run.smithi038.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.13.0-35-generic amd64 5.13.0-35.40~20.04.1 [18.3 MB] 2022-03-10T02:57:50.678 INFO:teuthology.orchestra.run.smithi195.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.13.0-35-generic amd64 5.13.0-35.40~20.04.1 [10.0 MB] 2022-03-10T02:57:51.112 INFO:teuthology.orchestra.run.smithi195.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.13.0-35-generic amd64 5.13.0-35.40~20.04.1 [51.1 MB] 2022-03-10T02:57:51.261 INFO:teuthology.orchestra.run.smithi037.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.13.0-35-generic amd64 5.13.0-35.40~20.04.1 [10.0 MB] 2022-03-10T02:57:51.353 INFO:teuthology.orchestra.run.smithi038.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.13.0-35-generic amd64 5.13.0-35.40~20.04.1 [10.0 MB] 2022-03-10T02:57:51.551 INFO:teuthology.orchestra.run.smithi037.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.13.0-35-generic amd64 5.13.0-35.40~20.04.1 [51.1 MB] 2022-03-10T02:57:51.675 INFO:teuthology.orchestra.run.smithi038.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.13.0-35-generic amd64 5.13.0-35.40~20.04.1 [51.1 MB] 2022-03-10T02:57:53.066 INFO:teuthology.orchestra.run.smithi195.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic-hwe-20.04 amd64 5.13.0.35.40~20.04.20 [2,632 B] 2022-03-10T02:57:53.066 INFO:teuthology.orchestra.run.smithi195.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libevdev2 amd64 1.9.0+dfsg-1ubuntu0.1 [31.6 kB] 2022-03-10T02:57:53.067 INFO:teuthology.orchestra.run.smithi195.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 thermald amd64 1.9.1-1ubuntu0.6 [233 kB] 2022-03-10T02:57:53.115 INFO:teuthology.orchestra.run.smithi195.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/main amd64 upower amd64 0.99.11-1build2 [104 kB] 2022-03-10T02:57:53.117 INFO:teuthology.orchestra.run.smithi195.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 usbmuxd amd64 1.1.1~git20191130.9af2b12-1 [38.4 kB] 2022-03-10T02:57:53.506 INFO:teuthology.orchestra.run.smithi037.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic-hwe-20.04 amd64 5.13.0.35.40~20.04.20 [2,632 B] 2022-03-10T02:57:53.506 INFO:teuthology.orchestra.run.smithi037.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libevdev2 amd64 1.9.0+dfsg-1ubuntu0.1 [31.6 kB] 2022-03-10T02:57:53.507 INFO:teuthology.orchestra.run.smithi037.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 thermald amd64 1.9.1-1ubuntu0.6 [233 kB] 2022-03-10T02:57:53.512 INFO:teuthology.orchestra.run.smithi037.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/main amd64 upower amd64 0.99.11-1build2 [104 kB] 2022-03-10T02:57:53.514 INFO:teuthology.orchestra.run.smithi037.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 usbmuxd amd64 1.1.1~git20191130.9af2b12-1 [38.4 kB] 2022-03-10T02:57:53.532 INFO:teuthology.orchestra.run.smithi195.stdout:Fetched 80.1 MB in 4s (20.7 MB/s) 2022-03-10T02:57:53.685 INFO:teuthology.orchestra.run.smithi038.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic-hwe-20.04 amd64 5.13.0.35.40~20.04.20 [2,632 B] 2022-03-10T02:57:53.686 INFO:teuthology.orchestra.run.smithi038.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libevdev2 amd64 1.9.0+dfsg-1ubuntu0.1 [31.6 kB] 2022-03-10T02:57:53.686 INFO:teuthology.orchestra.run.smithi038.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 thermald amd64 1.9.1-1ubuntu0.6 [233 kB] 2022-03-10T02:57:53.692 INFO:teuthology.orchestra.run.smithi038.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/main amd64 upower amd64 0.99.11-1build2 [104 kB] 2022-03-10T02:57:53.694 INFO:teuthology.orchestra.run.smithi038.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 usbmuxd amd64 1.1.1~git20191130.9af2b12-1 [38.4 kB] 2022-03-10T02:57:53.744 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package libdbus-glib-1-2:amd64. 2022-03-10T02:57:53.978 INFO:teuthology.orchestra.run.smithi037.stdout:Fetched 80.1 MB in 4s (19.8 MB/s) 2022-03-10T02:57:54.107 INFO:teuthology.orchestra.run.smithi038.stdout:Fetched 80.1 MB in 4s (19.9 MB/s) 2022-03-10T02:57:54.192 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package libdbus-glib-1-2:amd64. 2022-03-10T02:57:54.325 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package libdbus-glib-1-2:amd64. 2022-03-10T02:57:55.216 INFO:teuthology.orchestra.run.smithi195.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 102042 files and directories currently installed.) 2022-03-10T02:57:55.222 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../00-libdbus-glib-1-2_0.110-5fakssync1_amd64.deb ... 2022-03-10T02:57:55.366 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking libdbus-glib-1-2:amd64 (0.110-5fakssync1) ... 2022-03-10T02:57:55.656 INFO:teuthology.orchestra.run.smithi037.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 102042 files and directories currently installed.) 2022-03-10T02:57:55.660 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../00-libdbus-glib-1-2_0.110-5fakssync1_amd64.deb ... 2022-03-10T02:57:55.727 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package libplist3:amd64. 2022-03-10T02:57:55.743 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../01-libplist3_2.1.0-4build2_amd64.deb ... 2022-03-10T02:57:55.786 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking libplist3:amd64 (2.1.0-4build2) ... 2022-03-10T02:57:55.793 INFO:teuthology.orchestra.run.smithi038.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 102042 files and directories currently installed.) 2022-03-10T02:57:55.798 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../00-libdbus-glib-1-2_0.110-5fakssync1_amd64.deb ... 2022-03-10T02:57:55.841 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking libdbus-glib-1-2:amd64 (0.110-5fakssync1) ... 2022-03-10T02:57:55.947 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking libdbus-glib-1-2:amd64 (0.110-5fakssync1) ... 2022-03-10T02:57:56.080 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package libusbmuxd6:amd64. 2022-03-10T02:57:56.095 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../02-libusbmuxd6_2.0.1-2_amd64.deb ... 2022-03-10T02:57:56.130 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking libusbmuxd6:amd64 (2.0.1-2) ... 2022-03-10T02:57:56.235 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package libplist3:amd64. 2022-03-10T02:57:56.248 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../01-libplist3_2.1.0-4build2_amd64.deb ... 2022-03-10T02:57:56.285 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking libplist3:amd64 (2.1.0-4build2) ... 2022-03-10T02:57:56.467 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package libplist3:amd64. 2022-03-10T02:57:56.482 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../01-libplist3_2.1.0-4build2_amd64.deb ... 2022-03-10T02:57:56.500 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package libimobiledevice6:amd64. 2022-03-10T02:57:56.515 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../03-libimobiledevice6_1.2.1~git20191129.9f79242-1build1_amd64.deb ... 2022-03-10T02:57:56.524 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking libplist3:amd64 (2.1.0-4build2) ... 2022-03-10T02:57:56.558 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking libimobiledevice6:amd64 (1.2.1~git20191129.9f79242-1build1) ... 2022-03-10T02:57:56.696 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package libusbmuxd6:amd64. 2022-03-10T02:57:56.711 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../02-libusbmuxd6_2.0.1-2_amd64.deb ... 2022-03-10T02:57:56.754 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking libusbmuxd6:amd64 (2.0.1-2) ... 2022-03-10T02:57:56.878 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package libusbmuxd6:amd64. 2022-03-10T02:57:56.891 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../02-libusbmuxd6_2.0.1-2_amd64.deb ... 2022-03-10T02:57:56.894 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package libupower-glib3:amd64. 2022-03-10T02:57:56.908 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../04-libupower-glib3_0.99.11-1build2_amd64.deb ... 2022-03-10T02:57:56.927 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking libusbmuxd6:amd64 (2.0.1-2) ... 2022-03-10T02:57:56.969 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking libupower-glib3:amd64 (0.99.11-1build2) ... 2022-03-10T02:57:57.116 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package libimobiledevice6:amd64. 2022-03-10T02:57:57.129 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../03-libimobiledevice6_1.2.1~git20191129.9f79242-1build1_amd64.deb ... 2022-03-10T02:57:57.165 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking libimobiledevice6:amd64 (1.2.1~git20191129.9f79242-1build1) ... 2022-03-10T02:57:57.256 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package linux-modules-5.13.0-35-generic. 2022-03-10T02:57:57.264 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package libimobiledevice6:amd64. 2022-03-10T02:57:57.272 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../05-linux-modules-5.13.0-35-generic_5.13.0-35.40~20.04.1_amd64.deb ... 2022-03-10T02:57:57.278 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../03-libimobiledevice6_1.2.1~git20191129.9f79242-1build1_amd64.deb ... 2022-03-10T02:57:57.306 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking linux-modules-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-10T02:57:57.322 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking libimobiledevice6:amd64 (1.2.1~git20191129.9f79242-1build1) ... 2022-03-10T02:57:57.527 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package libupower-glib3:amd64. 2022-03-10T02:57:57.540 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../04-libupower-glib3_0.99.11-1build2_amd64.deb ... 2022-03-10T02:57:57.585 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking libupower-glib3:amd64 (0.99.11-1build2) ... 2022-03-10T02:57:57.725 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package libupower-glib3:amd64. 2022-03-10T02:57:57.739 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../04-libupower-glib3_0.99.11-1build2_amd64.deb ... 2022-03-10T02:57:57.775 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking libupower-glib3:amd64 (0.99.11-1build2) ... 2022-03-10T02:57:57.905 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package linux-modules-5.13.0-35-generic. 2022-03-10T02:57:57.917 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../05-linux-modules-5.13.0-35-generic_5.13.0-35.40~20.04.1_amd64.deb ... 2022-03-10T02:57:57.946 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking linux-modules-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-10T02:57:58.104 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package linux-modules-5.13.0-35-generic. 2022-03-10T02:57:58.117 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../05-linux-modules-5.13.0-35-generic_5.13.0-35.40~20.04.1_amd64.deb ... 2022-03-10T02:57:58.187 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking linux-modules-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-10T02:57:59.548 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package linux-image-5.13.0-35-generic. 2022-03-10T02:57:59.565 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../06-linux-image-5.13.0-35-generic_5.13.0-35.40~20.04.1_amd64.deb ... 2022-03-10T02:57:59.660 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking linux-image-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-10T02:58:00.143 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package linux-modules-extra-5.13.0-35-generic. 2022-03-10T02:58:00.156 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../07-linux-modules-extra-5.13.0-35-generic_5.13.0-35.40~20.04.1_amd64.deb ... 2022-03-10T02:58:00.193 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking linux-modules-extra-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-10T02:58:00.230 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package linux-image-5.13.0-35-generic. 2022-03-10T02:58:00.244 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../06-linux-image-5.13.0-35-generic_5.13.0-35.40~20.04.1_amd64.deb ... 2022-03-10T02:58:00.369 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking linux-image-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-10T02:58:00.506 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package linux-image-5.13.0-35-generic. 2022-03-10T02:58:00.519 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../06-linux-image-5.13.0-35-generic_5.13.0-35.40~20.04.1_amd64.deb ... 2022-03-10T02:58:00.610 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking linux-image-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-10T02:58:00.882 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package linux-modules-extra-5.13.0-35-generic. 2022-03-10T02:58:00.898 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../07-linux-modules-extra-5.13.0-35-generic_5.13.0-35.40~20.04.1_amd64.deb ... 2022-03-10T02:58:00.966 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking linux-modules-extra-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-10T02:58:01.109 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package linux-modules-extra-5.13.0-35-generic. 2022-03-10T02:58:01.124 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../07-linux-modules-extra-5.13.0-35-generic_5.13.0-35.40~20.04.1_amd64.deb ... 2022-03-10T02:58:01.167 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking linux-modules-extra-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-10T02:58:06.160 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package linux-image-generic-hwe-20.04. 2022-03-10T02:58:06.176 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../08-linux-image-generic-hwe-20.04_5.13.0.35.40~20.04.20_amd64.deb ... 2022-03-10T02:58:06.252 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking linux-image-generic-hwe-20.04 (5.13.0.35.40~20.04.20) ... 2022-03-10T02:58:06.563 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package libevdev2:amd64. 2022-03-10T02:58:06.576 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../09-libevdev2_1.9.0+dfsg-1ubuntu0.1_amd64.deb ... 2022-03-10T02:58:06.622 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking libevdev2:amd64 (1.9.0+dfsg-1ubuntu0.1) ... 2022-03-10T02:58:06.947 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package linux-image-generic-hwe-20.04. 2022-03-10T02:58:06.965 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../08-linux-image-generic-hwe-20.04_5.13.0.35.40~20.04.20_amd64.deb ... 2022-03-10T02:58:07.005 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking linux-image-generic-hwe-20.04 (5.13.0.35.40~20.04.20) ... 2022-03-10T02:58:07.008 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package thermald. 2022-03-10T02:58:07.021 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../10-thermald_1.9.1-1ubuntu0.6_amd64.deb ... 2022-03-10T02:58:07.025 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package linux-image-generic-hwe-20.04. 2022-03-10T02:58:07.042 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../08-linux-image-generic-hwe-20.04_5.13.0.35.40~20.04.20_amd64.deb ... 2022-03-10T02:58:07.058 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking thermald (1.9.1-1ubuntu0.6) ... 2022-03-10T02:58:07.075 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking linux-image-generic-hwe-20.04 (5.13.0.35.40~20.04.20) ... 2022-03-10T02:58:07.351 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package libevdev2:amd64. 2022-03-10T02:58:07.365 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../09-libevdev2_1.9.0+dfsg-1ubuntu0.1_amd64.deb ... 2022-03-10T02:58:07.395 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package libevdev2:amd64. 2022-03-10T02:58:07.400 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking libevdev2:amd64 (1.9.0+dfsg-1ubuntu0.1) ... 2022-03-10T02:58:07.412 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../09-libevdev2_1.9.0+dfsg-1ubuntu0.1_amd64.deb ... 2022-03-10T02:58:07.453 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking libevdev2:amd64 (1.9.0+dfsg-1ubuntu0.1) ... 2022-03-10T02:58:07.616 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package upower. 2022-03-10T02:58:07.629 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../11-upower_0.99.11-1build2_amd64.deb ... 2022-03-10T02:58:07.676 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking upower (0.99.11-1build2) ... 2022-03-10T02:58:07.812 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package thermald. 2022-03-10T02:58:07.828 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../10-thermald_1.9.1-1ubuntu0.6_amd64.deb ... 2022-03-10T02:58:07.832 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package thermald. 2022-03-10T02:58:07.846 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../10-thermald_1.9.1-1ubuntu0.6_amd64.deb ... 2022-03-10T02:58:07.879 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking thermald (1.9.1-1ubuntu0.6) ... 2022-03-10T02:58:07.890 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking thermald (1.9.1-1ubuntu0.6) ... 2022-03-10T02:58:08.033 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package usbmuxd. 2022-03-10T02:58:08.049 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../12-usbmuxd_1.1.1~git20191130.9af2b12-1_amd64.deb ... 2022-03-10T02:58:08.091 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking usbmuxd (1.1.1~git20191130.9af2b12-1) ... 2022-03-10T02:58:08.364 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up libplist3:amd64 (2.1.0-4build2) ... 2022-03-10T02:58:08.453 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package upower. 2022-03-10T02:58:08.470 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../11-upower_0.99.11-1build2_amd64.deb ... 2022-03-10T02:58:08.473 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package upower. 2022-03-10T02:58:08.487 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../11-upower_0.99.11-1build2_amd64.deb ... 2022-03-10T02:58:08.512 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up libdbus-glib-1-2:amd64 (0.110-5fakssync1) ... 2022-03-10T02:58:08.521 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking upower (0.99.11-1build2) ... 2022-03-10T02:58:08.542 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking upower (0.99.11-1build2) ... 2022-03-10T02:58:08.638 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up libusbmuxd6:amd64 (2.0.1-2) ... 2022-03-10T02:58:08.764 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up libupower-glib3:amd64 (0.99.11-1build2) ... 2022-03-10T02:58:08.890 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up libimobiledevice6:amd64 (1.2.1~git20191129.9f79242-1build1) ... 2022-03-10T02:58:08.953 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package usbmuxd. 2022-03-10T02:58:08.957 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package usbmuxd. 2022-03-10T02:58:08.968 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../12-usbmuxd_1.1.1~git20191130.9af2b12-1_amd64.deb ... 2022-03-10T02:58:08.971 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../12-usbmuxd_1.1.1~git20191130.9af2b12-1_amd64.deb ... 2022-03-10T02:58:09.011 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking usbmuxd (1.1.1~git20191130.9af2b12-1) ... 2022-03-10T02:58:09.015 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking usbmuxd (1.1.1~git20191130.9af2b12-1) ... 2022-03-10T02:58:09.033 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up libevdev2:amd64 (1.9.0+dfsg-1ubuntu0.1) ... 2022-03-10T02:58:09.159 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up upower (0.99.11-1build2) ... 2022-03-10T02:58:09.307 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up libplist3:amd64 (2.1.0-4build2) ... 2022-03-10T02:58:09.336 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up libplist3:amd64 (2.1.0-4build2) ... 2022-03-10T02:58:09.475 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up libdbus-glib-1-2:amd64 (0.110-5fakssync1) ... 2022-03-10T02:58:09.479 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up libdbus-glib-1-2:amd64 (0.110-5fakssync1) ... 2022-03-10T02:58:09.596 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up libusbmuxd6:amd64 (2.0.1-2) ... 2022-03-10T02:58:09.600 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up libusbmuxd6:amd64 (2.0.1-2) ... 2022-03-10T02:58:09.613 INFO:teuthology.orchestra.run.smithi195.stdout:upower.service is a disabled or a static unit, not starting it. 2022-03-10T02:58:09.660 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up usbmuxd (1.1.1~git20191130.9af2b12-1) ... 2022-03-10T02:58:09.723 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up libupower-glib3:amd64 (0.99.11-1build2) ... 2022-03-10T02:58:09.785 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up libupower-glib3:amd64 (0.99.11-1build2) ... 2022-03-10T02:58:09.827 INFO:teuthology.orchestra.run.smithi195.stdout:Warning: The home dir /var/lib/usbmux you specified can't be accessed: No such file or directory 2022-03-10T02:58:09.832 INFO:teuthology.orchestra.run.smithi195.stdout:Adding system user `usbmux' (UID 114) ... 2022-03-10T02:58:09.832 INFO:teuthology.orchestra.run.smithi195.stdout:Adding new user `usbmux' (UID 114) with group `plugdev' ... 2022-03-10T02:58:09.874 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up libimobiledevice6:amd64 (1.2.1~git20191129.9f79242-1build1) ... 2022-03-10T02:58:09.911 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up libimobiledevice6:amd64 (1.2.1~git20191129.9f79242-1build1) ... 2022-03-10T02:58:10.000 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up libevdev2:amd64 (1.9.0+dfsg-1ubuntu0.1) ... 2022-03-10T02:58:10.053 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up libevdev2:amd64 (1.9.0+dfsg-1ubuntu0.1) ... 2022-03-10T02:58:10.156 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up upower (0.99.11-1build2) ... 2022-03-10T02:58:10.163 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up upower (0.99.11-1build2) ... 2022-03-10T02:58:10.319 INFO:teuthology.orchestra.run.smithi195.stdout:Not creating home directory `/var/lib/usbmux'. 2022-03-10T02:58:10.534 INFO:teuthology.orchestra.run.smithi195.stdout:usbmuxd.service is a disabled or a static unit, not starting it. 2022-03-10T02:58:10.590 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up thermald (1.9.1-1ubuntu0.6) ... 2022-03-10T02:58:10.606 INFO:teuthology.orchestra.run.smithi038.stdout:upower.service is a disabled or a static unit, not starting it. 2022-03-10T02:58:10.624 INFO:teuthology.orchestra.run.smithi037.stdout:upower.service is a disabled or a static unit, not starting it. 2022-03-10T02:58:10.689 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up usbmuxd (1.1.1~git20191130.9af2b12-1) ... 2022-03-10T02:58:10.744 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up usbmuxd (1.1.1~git20191130.9af2b12-1) ... 2022-03-10T02:58:10.869 INFO:teuthology.orchestra.run.smithi037.stdout:Warning: The home dir /var/lib/usbmux you specified can't be accessed: No such file or directory 2022-03-10T02:58:10.876 INFO:teuthology.orchestra.run.smithi037.stdout:Adding system user `usbmux' (UID 114) ... 2022-03-10T02:58:10.876 INFO:teuthology.orchestra.run.smithi037.stdout:Adding new user `usbmux' (UID 114) with group `plugdev' ... 2022-03-10T02:58:10.902 INFO:teuthology.orchestra.run.smithi195.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.thermald.service -> /lib/systemd/system/thermald.service. 2022-03-10T02:58:10.902 INFO:teuthology.orchestra.run.smithi195.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/thermald.service -> /lib/systemd/system/thermald.service. 2022-03-10T02:58:10.924 INFO:teuthology.orchestra.run.smithi038.stdout:Warning: The home dir /var/lib/usbmux you specified can't be accessed: No such file or directory 2022-03-10T02:58:10.935 INFO:teuthology.orchestra.run.smithi038.stdout:Adding system user `usbmux' (UID 114) ... 2022-03-10T02:58:10.936 INFO:teuthology.orchestra.run.smithi038.stdout:Adding new user `usbmux' (UID 114) with group `plugdev' ... 2022-03-10T02:58:11.386 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up linux-image-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-10T02:58:11.398 INFO:teuthology.orchestra.run.smithi037.stdout:Not creating home directory `/var/lib/usbmux'. 2022-03-10T02:58:11.437 INFO:teuthology.orchestra.run.smithi038.stdout:Not creating home directory `/var/lib/usbmux'. 2022-03-10T02:58:11.619 INFO:teuthology.orchestra.run.smithi037.stdout:usbmuxd.service is a disabled or a static unit, not starting it. 2022-03-10T02:58:11.661 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up thermald (1.9.1-1ubuntu0.6) ... 2022-03-10T02:58:11.667 INFO:teuthology.orchestra.run.smithi038.stdout:usbmuxd.service is a disabled or a static unit, not starting it. 2022-03-10T02:58:11.833 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up thermald (1.9.1-1ubuntu0.6) ... 2022-03-10T02:58:11.949 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.thermald.service -> /lib/systemd/system/thermald.service. 2022-03-10T02:58:11.950 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/thermald.service -> /lib/systemd/system/thermald.service. 2022-03-10T02:58:12.187 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.thermald.service -> /lib/systemd/system/thermald.service. 2022-03-10T02:58:12.187 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/thermald.service -> /lib/systemd/system/thermald.service. 2022-03-10T02:58:12.417 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up linux-image-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-10T02:58:12.639 INFO:teuthology.orchestra.run.smithi195.stdout:I: /boot/vmlinuz is now a symlink to vmlinuz-5.13.0-35-generic 2022-03-10T02:58:12.640 INFO:teuthology.orchestra.run.smithi195.stdout:I: /boot/initrd.img is now a symlink to initrd.img-5.13.0-35-generic 2022-03-10T02:58:12.663 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up linux-image-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-10T02:58:12.813 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up linux-modules-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-10T02:58:13.699 INFO:teuthology.orchestra.run.smithi037.stdout:I: /boot/vmlinuz is now a symlink to vmlinuz-5.13.0-35-generic 2022-03-10T02:58:13.699 INFO:teuthology.orchestra.run.smithi037.stdout:I: /boot/initrd.img is now a symlink to initrd.img-5.13.0-35-generic 2022-03-10T02:58:13.869 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up linux-modules-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-10T02:58:13.949 INFO:teuthology.orchestra.run.smithi038.stdout:I: /boot/vmlinuz is now a symlink to vmlinuz-5.13.0-35-generic 2022-03-10T02:58:13.949 INFO:teuthology.orchestra.run.smithi038.stdout:I: /boot/initrd.img is now a symlink to initrd.img-5.13.0-35-generic 2022-03-10T02:58:14.124 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up linux-modules-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-10T02:58:14.148 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up linux-modules-extra-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-10T02:58:15.229 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up linux-modules-extra-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-10T02:58:15.483 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up linux-modules-extra-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-10T02:58:15.520 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up linux-image-generic-hwe-20.04 (5.13.0.35.40~20.04.20) ... 2022-03-10T02:58:16.572 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up linux-image-generic-hwe-20.04 (5.13.0.35.40~20.04.20) ... 2022-03-10T02:58:16.576 INFO:teuthology.orchestra.run.smithi195.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.2) ... 2022-03-10T02:58:16.724 INFO:teuthology.orchestra.run.smithi037.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.2) ... 2022-03-10T02:58:16.848 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up linux-image-generic-hwe-20.04 (5.13.0.35.40~20.04.20) ... 2022-03-10T02:58:16.979 INFO:teuthology.orchestra.run.smithi038.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.2) ... 2022-03-10T02:58:20.765 INFO:teuthology.orchestra.run.smithi195.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-03-10T02:58:21.492 INFO:teuthology.orchestra.run.smithi195.stdout:Processing triggers for dbus (1.12.16-2ubuntu2.1) ... 2022-03-10T02:58:21.586 INFO:teuthology.orchestra.run.smithi038.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-03-10T02:58:21.617 INFO:teuthology.orchestra.run.smithi195.stdout:Processing triggers for linux-image-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-10T02:58:21.801 INFO:teuthology.orchestra.run.smithi195.stdout:/etc/kernel/postinst.d/initramfs-tools: 2022-03-10T02:58:21.801 INFO:teuthology.orchestra.run.smithi195.stdout:update-initramfs: Generating /boot/initrd.img-5.13.0-35-generic 2022-03-10T02:58:21.820 INFO:teuthology.orchestra.run.smithi037.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-03-10T02:58:22.288 INFO:teuthology.orchestra.run.smithi038.stdout:Processing triggers for dbus (1.12.16-2ubuntu2.1) ... 2022-03-10T02:58:22.388 INFO:teuthology.orchestra.run.smithi038.stdout:Processing triggers for linux-image-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-10T02:58:22.439 INFO:teuthology.orchestra.run.smithi037.stdout:Processing triggers for dbus (1.12.16-2ubuntu2.1) ... 2022-03-10T02:58:22.539 INFO:teuthology.orchestra.run.smithi037.stdout:Processing triggers for linux-image-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-10T02:58:22.593 INFO:teuthology.orchestra.run.smithi038.stdout:/etc/kernel/postinst.d/initramfs-tools: 2022-03-10T02:58:22.594 INFO:teuthology.orchestra.run.smithi038.stdout:update-initramfs: Generating /boot/initrd.img-5.13.0-35-generic 2022-03-10T02:58:22.740 INFO:teuthology.orchestra.run.smithi037.stdout:/etc/kernel/postinst.d/initramfs-tools: 2022-03-10T02:58:22.741 INFO:teuthology.orchestra.run.smithi037.stdout:update-initramfs: Generating /boot/initrd.img-5.13.0-35-generic 2022-03-10T02:58:45.508 INFO:teuthology.orchestra.run.smithi195.stdout:/etc/kernel/postinst.d/zz-update-grub: 2022-03-10T02:58:45.508 INFO:teuthology.orchestra.run.smithi195.stdout:Sourcing file `/etc/default/grub' 2022-03-10T02:58:45.517 INFO:teuthology.orchestra.run.smithi195.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-03-10T02:58:45.536 INFO:teuthology.orchestra.run.smithi195.stdout:Generating grub configuration file ... 2022-03-10T02:58:45.867 INFO:teuthology.orchestra.run.smithi195.stdout:Found linux image: /boot/vmlinuz-5.13.0-35-generic 2022-03-10T02:58:45.895 INFO:teuthology.orchestra.run.smithi195.stdout:Found initrd image: /boot/initrd.img-5.13.0-35-generic 2022-03-10T02:58:46.306 INFO:teuthology.orchestra.run.smithi195.stdout:Found linux image: /boot/vmlinuz-5.4.0-97-generic 2022-03-10T02:58:46.318 INFO:teuthology.orchestra.run.smithi195.stdout:Found initrd image: /boot/initrd.img-5.4.0-97-generic 2022-03-10T02:58:46.644 INFO:teuthology.orchestra.run.smithi037.stdout:/etc/kernel/postinst.d/zz-update-grub: 2022-03-10T02:58:46.644 INFO:teuthology.orchestra.run.smithi037.stdout:Sourcing file `/etc/default/grub' 2022-03-10T02:58:46.653 INFO:teuthology.orchestra.run.smithi037.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-03-10T02:58:46.673 INFO:teuthology.orchestra.run.smithi037.stdout:Generating grub configuration file ... 2022-03-10T02:58:47.029 INFO:teuthology.orchestra.run.smithi037.stdout:Found linux image: /boot/vmlinuz-5.13.0-35-generic 2022-03-10T02:58:47.059 INFO:teuthology.orchestra.run.smithi037.stdout:Found initrd image: /boot/initrd.img-5.13.0-35-generic 2022-03-10T02:58:47.085 INFO:teuthology.orchestra.run.smithi038.stdout:/etc/kernel/postinst.d/zz-update-grub: 2022-03-10T02:58:47.085 INFO:teuthology.orchestra.run.smithi038.stdout:Sourcing file `/etc/default/grub' 2022-03-10T02:58:47.096 INFO:teuthology.orchestra.run.smithi038.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-03-10T02:58:47.113 INFO:teuthology.orchestra.run.smithi038.stdout:Generating grub configuration file ... 2022-03-10T02:58:47.466 INFO:teuthology.orchestra.run.smithi037.stdout:Found linux image: /boot/vmlinuz-5.4.0-97-generic 2022-03-10T02:58:47.476 INFO:teuthology.orchestra.run.smithi038.stdout:Found linux image: /boot/vmlinuz-5.13.0-35-generic 2022-03-10T02:58:47.478 INFO:teuthology.orchestra.run.smithi037.stdout:Found initrd image: /boot/initrd.img-5.4.0-97-generic 2022-03-10T02:58:47.511 INFO:teuthology.orchestra.run.smithi038.stdout:Found initrd image: /boot/initrd.img-5.13.0-35-generic 2022-03-10T02:58:47.956 INFO:teuthology.orchestra.run.smithi038.stdout:Found linux image: /boot/vmlinuz-5.4.0-97-generic 2022-03-10T02:58:47.968 INFO:teuthology.orchestra.run.smithi038.stdout:Found initrd image: /boot/initrd.img-5.4.0-97-generic 2022-03-10T02:58:48.729 INFO:teuthology.orchestra.run.smithi195.stdout:done 2022-03-10T02:58:49.078 DEBUG:teuthology.orchestra.run.smithi195:> dpkg -s linux-image-generic-hwe-20.04 2022-03-10T02:58:49.112 INFO:teuthology.orchestra.run.smithi195.stdout:Package: linux-image-generic-hwe-20.04 2022-03-10T02:58:49.112 INFO:teuthology.orchestra.run.smithi195.stdout:Status: install ok installed 2022-03-10T02:58:49.112 INFO:teuthology.orchestra.run.smithi195.stdout:Priority: optional 2022-03-10T02:58:49.113 INFO:teuthology.orchestra.run.smithi195.stdout:Section: kernel 2022-03-10T02:58:49.113 INFO:teuthology.orchestra.run.smithi195.stdout:Installed-Size: 19 2022-03-10T02:58:49.113 INFO:teuthology.orchestra.run.smithi195.stdout:Maintainer: Ubuntu Kernel Team 2022-03-10T02:58:49.114 INFO:teuthology.orchestra.run.smithi195.stdout:Architecture: amd64 2022-03-10T02:58:49.114 INFO:teuthology.orchestra.run.smithi195.stdout:Source: linux-meta-hwe-5.13 2022-03-10T02:58:49.114 INFO:teuthology.orchestra.run.smithi195.stdout:Version: 5.13.0.35.40~20.04.20 2022-03-10T02:58:49.114 INFO:teuthology.orchestra.run.smithi195.stdout:Provides: virtualbox-guest-modules (= 5.13.0-35), wireguard-modules (= 1.0.0), zfs-modules (= 2.0.6-1ubuntu2) 2022-03-10T02:58:49.115 INFO:teuthology.orchestra.run.smithi195.stdout:Depends: linux-image-5.13.0-35-generic, linux-modules-extra-5.13.0-35-generic, linux-firmware, intel-microcode, amd64-microcode 2022-03-10T02:58:49.115 INFO:teuthology.orchestra.run.smithi195.stdout:Recommends: thermald 2022-03-10T02:58:49.115 INFO:teuthology.orchestra.run.smithi195.stdout:Description: Generic Linux kernel image 2022-03-10T02:58:49.115 INFO:teuthology.orchestra.run.smithi195.stdout: This package will always depend on the latest generic kernel image 2022-03-10T02:58:49.115 INFO:teuthology.orchestra.run.smithi195.stdout: available. 2022-03-10T02:58:49.116 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-97-generic Expected: 5.13.0-35-generic 2022-03-10T02:58:49.116 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2022-03-10T02:58:49.116 DEBUG:teuthology.task.kernel:src is distro, skipping download 2022-03-10T02:58:49.117 INFO:teuthology.task.kernel:Installing distro kernel on host.c... 2022-03-10T02:58:49.117 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi195.front.sepia.ceph.com, path=None, version=distro) 2022-03-10T02:58:49.117 DEBUG:teuthology.orchestra.run.smithi195:> sudo apt-get clean 2022-03-10T02:58:49.216 DEBUG:teuthology.orchestra.run.smithi195:> sudo apt-get update 2022-03-10T02:58:49.273 INFO:teuthology.orchestra.run.smithi037.stdout:done 2022-03-10T02:58:49.355 INFO:teuthology.orchestra.run.smithi195.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-03-10T02:58:49.468 INFO:teuthology.orchestra.run.smithi195.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-10T02:58:49.560 INFO:teuthology.orchestra.run.smithi195.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-03-10T02:58:49.648 INFO:teuthology.orchestra.run.smithi195.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-03-10T02:58:49.662 DEBUG:teuthology.orchestra.run.smithi037:> dpkg -s linux-image-generic-hwe-20.04 2022-03-10T02:58:49.702 INFO:teuthology.orchestra.run.smithi037.stdout:Package: linux-image-generic-hwe-20.04 2022-03-10T02:58:49.703 INFO:teuthology.orchestra.run.smithi037.stdout:Status: install ok installed 2022-03-10T02:58:49.703 INFO:teuthology.orchestra.run.smithi037.stdout:Priority: optional 2022-03-10T02:58:49.703 INFO:teuthology.orchestra.run.smithi037.stdout:Section: kernel 2022-03-10T02:58:49.704 INFO:teuthology.orchestra.run.smithi037.stdout:Installed-Size: 19 2022-03-10T02:58:49.704 INFO:teuthology.orchestra.run.smithi037.stdout:Maintainer: Ubuntu Kernel Team 2022-03-10T02:58:49.704 INFO:teuthology.orchestra.run.smithi037.stdout:Architecture: amd64 2022-03-10T02:58:49.704 INFO:teuthology.orchestra.run.smithi037.stdout:Source: linux-meta-hwe-5.13 2022-03-10T02:58:49.705 INFO:teuthology.orchestra.run.smithi037.stdout:Version: 5.13.0.35.40~20.04.20 2022-03-10T02:58:49.705 INFO:teuthology.orchestra.run.smithi037.stdout:Provides: virtualbox-guest-modules (= 5.13.0-35), wireguard-modules (= 1.0.0), zfs-modules (= 2.0.6-1ubuntu2) 2022-03-10T02:58:49.705 INFO:teuthology.orchestra.run.smithi037.stdout:Depends: linux-image-5.13.0-35-generic, linux-modules-extra-5.13.0-35-generic, linux-firmware, intel-microcode, amd64-microcode 2022-03-10T02:58:49.706 INFO:teuthology.orchestra.run.smithi037.stdout:Recommends: thermald 2022-03-10T02:58:49.706 INFO:teuthology.orchestra.run.smithi037.stdout:Description: Generic Linux kernel image 2022-03-10T02:58:49.706 INFO:teuthology.orchestra.run.smithi037.stdout: This package will always depend on the latest generic kernel image 2022-03-10T02:58:49.706 INFO:teuthology.orchestra.run.smithi037.stdout: available. 2022-03-10T02:58:49.707 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-97-generic Expected: 5.13.0-35-generic 2022-03-10T02:58:49.707 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2022-03-10T02:58:49.708 DEBUG:teuthology.task.kernel:src is distro, skipping download 2022-03-10T02:58:49.708 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2022-03-10T02:58:49.708 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi037.front.sepia.ceph.com, path=None, version=distro) 2022-03-10T02:58:49.709 DEBUG:teuthology.orchestra.run.smithi037:> sudo apt-get clean 2022-03-10T02:58:49.786 DEBUG:teuthology.orchestra.run.smithi037:> sudo apt-get update 2022-03-10T02:58:50.064 INFO:teuthology.orchestra.run.smithi037.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-10T02:58:50.065 INFO:teuthology.orchestra.run.smithi037.stdout:Hit:2 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-03-10T02:58:50.135 INFO:teuthology.orchestra.run.smithi037.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-03-10T02:58:50.217 INFO:teuthology.orchestra.run.smithi037.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-03-10T02:58:50.445 INFO:teuthology.orchestra.run.smithi038.stdout:done 2022-03-10T02:58:50.806 DEBUG:teuthology.orchestra.run.smithi038:> dpkg -s linux-image-generic-hwe-20.04 2022-03-10T02:58:50.838 INFO:teuthology.orchestra.run.smithi038.stdout:Package: linux-image-generic-hwe-20.04 2022-03-10T02:58:50.839 INFO:teuthology.orchestra.run.smithi038.stdout:Status: install ok installed 2022-03-10T02:58:50.839 INFO:teuthology.orchestra.run.smithi038.stdout:Priority: optional 2022-03-10T02:58:50.839 INFO:teuthology.orchestra.run.smithi038.stdout:Section: kernel 2022-03-10T02:58:50.839 INFO:teuthology.orchestra.run.smithi038.stdout:Installed-Size: 19 2022-03-10T02:58:50.840 INFO:teuthology.orchestra.run.smithi038.stdout:Maintainer: Ubuntu Kernel Team 2022-03-10T02:58:50.840 INFO:teuthology.orchestra.run.smithi038.stdout:Architecture: amd64 2022-03-10T02:58:50.840 INFO:teuthology.orchestra.run.smithi038.stdout:Source: linux-meta-hwe-5.13 2022-03-10T02:58:50.840 INFO:teuthology.orchestra.run.smithi038.stdout:Version: 5.13.0.35.40~20.04.20 2022-03-10T02:58:50.840 INFO:teuthology.orchestra.run.smithi038.stdout:Provides: virtualbox-guest-modules (= 5.13.0-35), wireguard-modules (= 1.0.0), zfs-modules (= 2.0.6-1ubuntu2) 2022-03-10T02:58:50.841 INFO:teuthology.orchestra.run.smithi038.stdout:Depends: linux-image-5.13.0-35-generic, linux-modules-extra-5.13.0-35-generic, linux-firmware, intel-microcode, amd64-microcode 2022-03-10T02:58:50.841 INFO:teuthology.orchestra.run.smithi038.stdout:Recommends: thermald 2022-03-10T02:58:50.841 INFO:teuthology.orchestra.run.smithi038.stdout:Description: Generic Linux kernel image 2022-03-10T02:58:50.841 INFO:teuthology.orchestra.run.smithi038.stdout: This package will always depend on the latest generic kernel image 2022-03-10T02:58:50.842 INFO:teuthology.orchestra.run.smithi038.stdout: available. 2022-03-10T02:58:50.842 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-97-generic Expected: 5.13.0-35-generic 2022-03-10T02:58:50.842 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2022-03-10T02:58:50.842 DEBUG:teuthology.task.kernel:src is distro, skipping download 2022-03-10T02:58:50.843 INFO:teuthology.task.kernel:Installing distro kernel on host.b... 2022-03-10T02:58:50.843 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi038.front.sepia.ceph.com, path=None, version=distro) 2022-03-10T02:58:50.843 DEBUG:teuthology.orchestra.run.smithi038:> sudo apt-get clean 2022-03-10T02:58:50.874 INFO:teuthology.orchestra.run.smithi195.stdout:Reading package lists... 2022-03-10T02:58:50.898 DEBUG:teuthology.orchestra.run.smithi195:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic-hwe-20.04 2022-03-10T02:58:50.946 DEBUG:teuthology.orchestra.run.smithi038:> sudo apt-get update 2022-03-10T02:58:50.966 INFO:teuthology.orchestra.run.smithi195.stdout:Reading package lists... 2022-03-10T02:58:51.120 INFO:teuthology.orchestra.run.smithi195.stdout:Building dependency tree... 2022-03-10T02:58:51.120 INFO:teuthology.orchestra.run.smithi195.stdout:Reading state information... 2022-03-10T02:58:51.227 INFO:teuthology.orchestra.run.smithi038.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-10T02:58:51.228 INFO:teuthology.orchestra.run.smithi038.stdout:Hit:2 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-03-10T02:58:51.287 INFO:teuthology.orchestra.run.smithi195.stdout:linux-image-generic-hwe-20.04 is already the newest version (5.13.0.35.40~20.04.20). 2022-03-10T02:58:51.287 INFO:teuthology.orchestra.run.smithi195.stdout:0 upgraded, 0 newly installed, 0 to remove and 94 not upgraded. 2022-03-10T02:58:51.289 DEBUG:teuthology.orchestra.run.smithi195:> dpkg -s linux-image-generic-hwe-20.04 2022-03-10T02:58:51.298 INFO:teuthology.orchestra.run.smithi038.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-03-10T02:58:51.310 INFO:teuthology.orchestra.run.smithi195.stdout:Package: linux-image-generic-hwe-20.04 2022-03-10T02:58:51.310 INFO:teuthology.orchestra.run.smithi195.stdout:Status: install ok installed 2022-03-10T02:58:51.310 INFO:teuthology.orchestra.run.smithi195.stdout:Priority: optional 2022-03-10T02:58:51.311 INFO:teuthology.orchestra.run.smithi195.stdout:Section: kernel 2022-03-10T02:58:51.311 INFO:teuthology.orchestra.run.smithi195.stdout:Installed-Size: 19 2022-03-10T02:58:51.311 INFO:teuthology.orchestra.run.smithi195.stdout:Maintainer: Ubuntu Kernel Team 2022-03-10T02:58:51.311 INFO:teuthology.orchestra.run.smithi195.stdout:Architecture: amd64 2022-03-10T02:58:51.312 INFO:teuthology.orchestra.run.smithi195.stdout:Source: linux-meta-hwe-5.13 2022-03-10T02:58:51.312 INFO:teuthology.orchestra.run.smithi195.stdout:Version: 5.13.0.35.40~20.04.20 2022-03-10T02:58:51.312 INFO:teuthology.orchestra.run.smithi195.stdout:Provides: virtualbox-guest-modules (= 5.13.0-35), wireguard-modules (= 1.0.0), zfs-modules (= 2.0.6-1ubuntu2) 2022-03-10T02:58:51.312 INFO:teuthology.orchestra.run.smithi195.stdout:Depends: linux-image-5.13.0-35-generic, linux-modules-extra-5.13.0-35-generic, linux-firmware, intel-microcode, amd64-microcode 2022-03-10T02:58:51.312 INFO:teuthology.orchestra.run.smithi195.stdout:Recommends: thermald 2022-03-10T02:58:51.313 INFO:teuthology.orchestra.run.smithi195.stdout:Description: Generic Linux kernel image 2022-03-10T02:58:51.313 INFO:teuthology.orchestra.run.smithi195.stdout: This package will always depend on the latest generic kernel image 2022-03-10T02:58:51.313 INFO:teuthology.orchestra.run.smithi195.stdout: available. 2022-03-10T02:58:51.314 DEBUG:teuthology.orchestra.run.smithi195:> mktemp 2022-03-10T02:58:51.360 INFO:teuthology.orchestra.run.smithi195.stdout:/tmp/tmp.T71byqOpNO 2022-03-10T02:58:51.360 DEBUG:teuthology.orchestra.run.smithi195:> sudo cp /boot/grub/grub.cfg /tmp/tmp.T71byqOpNO 2022-03-10T02:58:51.372 DEBUG:teuthology.orchestra.run.smithi195:> sudo chmod 0666 /tmp/tmp.T71byqOpNO 2022-03-10T02:58:51.387 INFO:teuthology.orchestra.run.smithi038.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-03-10T02:58:51.411 DEBUG:teuthology.orchestra.remote:smithi195:/tmp/tmp.T71byqOpNO is 8KB 2022-03-10T02:58:51.425 DEBUG:teuthology.orchestra.run.smithi195:> rm -fr /tmp/tmp.T71byqOpNO 2022-03-10T02:58:51.432 DEBUG:teuthology.orchestra.run.smithi195:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2022-03-10T02:58:51.489 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2022-03-10T02:58:51.489 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd of=/etc/grub.d/01_ceph_kernel 2022-03-10T02:58:51.489 DEBUG:teuthology.orchestra.run.smithi195:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2022-03-10T02:58:51.525 INFO:teuthology.task.kernel:Distro Kernel Version: 5.13.0-35-generic 2022-03-10T02:58:51.526 DEBUG:teuthology.orchestra.run.smithi195:> sudo update-grub 2022-03-10T02:58:51.653 INFO:teuthology.orchestra.run.smithi037.stdout:Reading package lists... 2022-03-10T02:58:51.675 DEBUG:teuthology.orchestra.run.smithi037:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic-hwe-20.04 2022-03-10T02:58:51.742 INFO:teuthology.orchestra.run.smithi037.stdout:Reading package lists... 2022-03-10T02:58:51.916 INFO:teuthology.orchestra.run.smithi037.stdout:Building dependency tree... 2022-03-10T02:58:51.916 INFO:teuthology.orchestra.run.smithi037.stdout:Reading state information... 2022-03-10T02:58:52.095 INFO:teuthology.orchestra.run.smithi037.stdout:linux-image-generic-hwe-20.04 is already the newest version (5.13.0.35.40~20.04.20). 2022-03-10T02:58:52.096 INFO:teuthology.orchestra.run.smithi037.stdout:0 upgraded, 0 newly installed, 0 to remove and 94 not upgraded. 2022-03-10T02:58:52.098 DEBUG:teuthology.orchestra.run.smithi037:> dpkg -s linux-image-generic-hwe-20.04 2022-03-10T02:58:52.119 INFO:teuthology.orchestra.run.smithi037.stdout:Package: linux-image-generic-hwe-20.04 2022-03-10T02:58:52.119 INFO:teuthology.orchestra.run.smithi037.stdout:Status: install ok installed 2022-03-10T02:58:52.119 INFO:teuthology.orchestra.run.smithi037.stdout:Priority: optional 2022-03-10T02:58:52.119 INFO:teuthology.orchestra.run.smithi037.stdout:Section: kernel 2022-03-10T02:58:52.119 INFO:teuthology.orchestra.run.smithi037.stdout:Installed-Size: 19 2022-03-10T02:58:52.120 INFO:teuthology.orchestra.run.smithi037.stdout:Maintainer: Ubuntu Kernel Team 2022-03-10T02:58:52.120 INFO:teuthology.orchestra.run.smithi037.stdout:Architecture: amd64 2022-03-10T02:58:52.120 INFO:teuthology.orchestra.run.smithi037.stdout:Source: linux-meta-hwe-5.13 2022-03-10T02:58:52.120 INFO:teuthology.orchestra.run.smithi037.stdout:Version: 5.13.0.35.40~20.04.20 2022-03-10T02:58:52.121 INFO:teuthology.orchestra.run.smithi037.stdout:Provides: virtualbox-guest-modules (= 5.13.0-35), wireguard-modules (= 1.0.0), zfs-modules (= 2.0.6-1ubuntu2) 2022-03-10T02:58:52.121 INFO:teuthology.orchestra.run.smithi037.stdout:Depends: linux-image-5.13.0-35-generic, linux-modules-extra-5.13.0-35-generic, linux-firmware, intel-microcode, amd64-microcode 2022-03-10T02:58:52.121 INFO:teuthology.orchestra.run.smithi037.stdout:Recommends: thermald 2022-03-10T02:58:52.121 INFO:teuthology.orchestra.run.smithi037.stdout:Description: Generic Linux kernel image 2022-03-10T02:58:52.121 INFO:teuthology.orchestra.run.smithi037.stdout: This package will always depend on the latest generic kernel image 2022-03-10T02:58:52.122 INFO:teuthology.orchestra.run.smithi037.stdout: available. 2022-03-10T02:58:52.122 DEBUG:teuthology.orchestra.run.smithi037:> mktemp 2022-03-10T02:58:52.126 INFO:teuthology.orchestra.run.smithi037.stdout:/tmp/tmp.7kHueJknTh 2022-03-10T02:58:52.127 DEBUG:teuthology.orchestra.run.smithi037:> sudo cp /boot/grub/grub.cfg /tmp/tmp.7kHueJknTh 2022-03-10T02:58:52.183 DEBUG:teuthology.orchestra.run.smithi037:> sudo chmod 0666 /tmp/tmp.7kHueJknTh 2022-03-10T02:58:52.224 DEBUG:teuthology.orchestra.remote:smithi037:/tmp/tmp.7kHueJknTh is 8KB 2022-03-10T02:58:52.239 DEBUG:teuthology.orchestra.run.smithi037:> rm -fr /tmp/tmp.7kHueJknTh 2022-03-10T02:58:52.246 DEBUG:teuthology.orchestra.run.smithi037:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2022-03-10T02:58:52.302 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2022-03-10T02:58:52.303 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd of=/etc/grub.d/01_ceph_kernel 2022-03-10T02:58:52.303 DEBUG:teuthology.orchestra.run.smithi037:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2022-03-10T02:58:52.334 INFO:teuthology.task.kernel:Distro Kernel Version: 5.13.0-35-generic 2022-03-10T02:58:52.335 DEBUG:teuthology.orchestra.run.smithi037:> sudo update-grub 2022-03-10T02:58:52.695 INFO:teuthology.orchestra.run.smithi195.stderr:Sourcing file `/etc/default/grub' 2022-03-10T02:58:52.697 INFO:teuthology.orchestra.run.smithi195.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-03-10T02:58:52.727 INFO:teuthology.orchestra.run.smithi195.stderr:Generating grub configuration file ... 2022-03-10T02:58:52.820 INFO:teuthology.orchestra.run.smithi038.stdout:Reading package lists... 2022-03-10T02:58:52.843 DEBUG:teuthology.orchestra.run.smithi038:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic-hwe-20.04 2022-03-10T02:58:52.909 INFO:teuthology.orchestra.run.smithi038.stdout:Reading package lists... 2022-03-10T02:58:52.960 INFO:teuthology.orchestra.run.smithi195.stderr:Found linux image: /boot/vmlinuz-5.13.0-35-generic 2022-03-10T02:58:52.972 INFO:teuthology.orchestra.run.smithi195.stderr:Found initrd image: /boot/initrd.img-5.13.0-35-generic 2022-03-10T02:58:53.089 INFO:teuthology.orchestra.run.smithi038.stdout:Building dependency tree... 2022-03-10T02:58:53.090 INFO:teuthology.orchestra.run.smithi038.stdout:Reading state information... 2022-03-10T02:58:53.287 INFO:teuthology.orchestra.run.smithi038.stdout:linux-image-generic-hwe-20.04 is already the newest version (5.13.0.35.40~20.04.20). 2022-03-10T02:58:53.287 INFO:teuthology.orchestra.run.smithi038.stdout:0 upgraded, 0 newly installed, 0 to remove and 94 not upgraded. 2022-03-10T02:58:53.289 DEBUG:teuthology.orchestra.run.smithi038:> dpkg -s linux-image-generic-hwe-20.04 2022-03-10T02:58:53.312 INFO:teuthology.orchestra.run.smithi038.stdout:Package: linux-image-generic-hwe-20.04 2022-03-10T02:58:53.312 INFO:teuthology.orchestra.run.smithi038.stdout:Status: install ok installed 2022-03-10T02:58:53.312 INFO:teuthology.orchestra.run.smithi038.stdout:Priority: optional 2022-03-10T02:58:53.312 INFO:teuthology.orchestra.run.smithi038.stdout:Section: kernel 2022-03-10T02:58:53.313 INFO:teuthology.orchestra.run.smithi038.stdout:Installed-Size: 19 2022-03-10T02:58:53.313 INFO:teuthology.orchestra.run.smithi038.stdout:Maintainer: Ubuntu Kernel Team 2022-03-10T02:58:53.313 INFO:teuthology.orchestra.run.smithi038.stdout:Architecture: amd64 2022-03-10T02:58:53.313 INFO:teuthology.orchestra.run.smithi038.stdout:Source: linux-meta-hwe-5.13 2022-03-10T02:58:53.314 INFO:teuthology.orchestra.run.smithi038.stdout:Version: 5.13.0.35.40~20.04.20 2022-03-10T02:58:53.314 INFO:teuthology.orchestra.run.smithi038.stdout:Provides: virtualbox-guest-modules (= 5.13.0-35), wireguard-modules (= 1.0.0), zfs-modules (= 2.0.6-1ubuntu2) 2022-03-10T02:58:53.314 INFO:teuthology.orchestra.run.smithi038.stdout:Depends: linux-image-5.13.0-35-generic, linux-modules-extra-5.13.0-35-generic, linux-firmware, intel-microcode, amd64-microcode 2022-03-10T02:58:53.314 INFO:teuthology.orchestra.run.smithi038.stdout:Recommends: thermald 2022-03-10T02:58:53.315 INFO:teuthology.orchestra.run.smithi038.stdout:Description: Generic Linux kernel image 2022-03-10T02:58:53.315 INFO:teuthology.orchestra.run.smithi038.stdout: This package will always depend on the latest generic kernel image 2022-03-10T02:58:53.315 INFO:teuthology.orchestra.run.smithi038.stdout: available. 2022-03-10T02:58:53.315 DEBUG:teuthology.orchestra.run.smithi038:> mktemp 2022-03-10T02:58:53.320 INFO:teuthology.orchestra.run.smithi195.stderr:Found linux image: /boot/vmlinuz-5.4.0-97-generic 2022-03-10T02:58:53.332 INFO:teuthology.orchestra.run.smithi195.stderr:Found initrd image: /boot/initrd.img-5.4.0-97-generic 2022-03-10T02:58:53.359 INFO:teuthology.orchestra.run.smithi038.stdout:/tmp/tmp.xOxYzoydHU 2022-03-10T02:58:53.359 DEBUG:teuthology.orchestra.run.smithi038:> sudo cp /boot/grub/grub.cfg /tmp/tmp.xOxYzoydHU 2022-03-10T02:58:53.375 DEBUG:teuthology.orchestra.run.smithi038:> sudo chmod 0666 /tmp/tmp.xOxYzoydHU 2022-03-10T02:58:53.463 DEBUG:teuthology.orchestra.remote:smithi038:/tmp/tmp.xOxYzoydHU is 8KB 2022-03-10T02:58:53.479 DEBUG:teuthology.orchestra.run.smithi038:> rm -fr /tmp/tmp.xOxYzoydHU 2022-03-10T02:58:53.487 DEBUG:teuthology.orchestra.run.smithi038:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2022-03-10T02:58:53.544 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2022-03-10T02:58:53.544 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/etc/grub.d/01_ceph_kernel 2022-03-10T02:58:53.544 DEBUG:teuthology.orchestra.run.smithi038:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2022-03-10T02:58:53.580 INFO:teuthology.task.kernel:Distro Kernel Version: 5.13.0-35-generic 2022-03-10T02:58:53.580 DEBUG:teuthology.orchestra.run.smithi038:> sudo update-grub 2022-03-10T02:58:53.620 INFO:teuthology.orchestra.run.smithi037.stderr:Sourcing file `/etc/default/grub' 2022-03-10T02:58:53.621 INFO:teuthology.orchestra.run.smithi037.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-03-10T02:58:53.638 INFO:teuthology.orchestra.run.smithi037.stderr:Generating grub configuration file ... 2022-03-10T02:58:53.880 INFO:teuthology.orchestra.run.smithi037.stderr:Found linux image: /boot/vmlinuz-5.13.0-35-generic 2022-03-10T02:58:53.891 INFO:teuthology.orchestra.run.smithi037.stderr:Found initrd image: /boot/initrd.img-5.13.0-35-generic 2022-03-10T02:58:54.257 INFO:teuthology.orchestra.run.smithi037.stderr:Found linux image: /boot/vmlinuz-5.4.0-97-generic 2022-03-10T02:58:54.268 INFO:teuthology.orchestra.run.smithi037.stderr:Found initrd image: /boot/initrd.img-5.4.0-97-generic 2022-03-10T02:58:54.674 INFO:teuthology.orchestra.run.smithi195.stderr:done 2022-03-10T02:58:54.675 DEBUG:teuthology.orchestra.run.smithi195:> sudo shutdown -r now 2022-03-10T02:58:54.968 INFO:teuthology.orchestra.run.smithi037.stderr:done 2022-03-10T02:58:54.970 DEBUG:teuthology.orchestra.run.smithi037:> sudo shutdown -r now 2022-03-10T02:58:55.004 INFO:teuthology.orchestra.run.smithi038.stderr:Sourcing file `/etc/default/grub' 2022-03-10T02:58:55.007 INFO:teuthology.orchestra.run.smithi038.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-03-10T02:58:55.028 INFO:teuthology.orchestra.run.smithi038.stderr:Generating grub configuration file ... 2022-03-10T02:58:55.278 INFO:teuthology.orchestra.run.smithi038.stderr:Found linux image: /boot/vmlinuz-5.13.0-35-generic 2022-03-10T02:58:55.288 INFO:teuthology.orchestra.run.smithi038.stderr:Found initrd image: /boot/initrd.img-5.13.0-35-generic 2022-03-10T02:58:55.686 INFO:teuthology.orchestra.run.smithi038.stderr:Found linux image: /boot/vmlinuz-5.4.0-97-generic 2022-03-10T02:58:55.698 INFO:teuthology.orchestra.run.smithi038.stderr:Found initrd image: /boot/initrd.img-5.4.0-97-generic 2022-03-10T02:58:57.075 INFO:teuthology.orchestra.run.smithi038.stderr:done 2022-03-10T02:58:57.076 DEBUG:teuthology.orchestra.run.smithi038:> sudo shutdown -r now 2022-03-10T02:59:24.697 INFO:teuthology.task.kernel:Checking client host.c for new kernel version... 2022-03-10T02:59:24.698 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi195.front.sepia.ceph.com' 2022-03-10T02:59:24.699 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi195.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-10T02:59:24.973 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2022-03-10T02:59:24.974 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2022-03-10T02:59:24.974 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi037.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-10T02:59:27.082 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2022-03-10T02:59:27.082 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi038.front.sepia.ceph.com' 2022-03-10T02:59:27.083 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi038.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-10T02:59:43.179 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.195 2022-03-10T02:59:43.439 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.37 2022-03-10T02:59:45.483 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.38 2022-03-10T03:00:13.208 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi195.front.sepia.ceph.com' 2022-03-10T03:00:13.209 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi195.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-10T03:00:13.440 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2022-03-10T03:00:13.441 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi037.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-10T03:00:15.486 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi038.front.sepia.ceph.com' 2022-03-10T03:00:15.487 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi038.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-10T03:00:28.907 DEBUG:teuthology.orchestra.run.smithi195:> true 2022-03-10T03:00:28.965 DEBUG:teuthology.orchestra.run.smithi037:> true 2022-03-10T03:00:29.454 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi037.front.sepia.ceph.com' 2022-03-10T03:00:29.455 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "5.13.0-35-generic"... 2022-03-10T03:00:29.455 DEBUG:teuthology.orchestra.run.smithi037:> uname -r 2022-03-10T03:00:29.462 INFO:teuthology.orchestra.run.smithi037.stdout:5.13.0-35-generic 2022-03-10T03:00:29.463 DEBUG:teuthology.task.kernel:current kernel version is 5.13.0-35-generic vs 5.13.0-35-generic 2022-03-10T03:00:29.463 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2022-03-10T03:00:29.463 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2022-03-10T03:00:29.865 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi195.front.sepia.ceph.com' 2022-03-10T03:00:29.865 INFO:teuthology.task.kernel:Checking kernel version of host.c, want "5.13.0-35-generic"... 2022-03-10T03:00:29.866 DEBUG:teuthology.orchestra.run.smithi195:> uname -r 2022-03-10T03:00:29.913 INFO:teuthology.orchestra.run.smithi195.stdout:5.13.0-35-generic 2022-03-10T03:00:29.913 DEBUG:teuthology.task.kernel:current kernel version is 5.13.0-35-generic vs 5.13.0-35-generic 2022-03-10T03:00:29.914 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2022-03-10T03:00:29.914 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2022-03-10T03:00:30.464 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2022-03-10T03:00:30.465 DEBUG:teuthology.orchestra.run.smithi037:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2022-03-10T03:00:30.527 INFO:teuthology.orchestra.run.smithi037.stdout:ttyS1 2022-03-10T03:00:30.553 DEBUG:teuthology.parallel:result is None 2022-03-10T03:00:30.915 INFO:teuthology.task.kernel:Enabling kdb on host.c... 2022-03-10T03:00:30.915 DEBUG:teuthology.orchestra.run.smithi195:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2022-03-10T03:00:30.998 INFO:teuthology.orchestra.run.smithi195.stdout:ttyS1 2022-03-10T03:00:31.017 DEBUG:teuthology.parallel:result is None 2022-03-10T03:00:31.394 DEBUG:teuthology.orchestra.run.smithi038:> true 2022-03-10T03:00:32.268 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi038.front.sepia.ceph.com' 2022-03-10T03:00:32.269 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "5.13.0-35-generic"... 2022-03-10T03:00:32.269 DEBUG:teuthology.orchestra.run.smithi038:> uname -r 2022-03-10T03:00:32.274 INFO:teuthology.orchestra.run.smithi038.stdout:5.13.0-35-generic 2022-03-10T03:00:32.274 DEBUG:teuthology.task.kernel:current kernel version is 5.13.0-35-generic vs 5.13.0-35-generic 2022-03-10T03:00:32.275 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2022-03-10T03:00:32.275 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2022-03-10T03:00:33.276 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2022-03-10T03:00:33.276 DEBUG:teuthology.orchestra.run.smithi038:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2022-03-10T03:00:33.416 INFO:teuthology.orchestra.run.smithi038.stdout:ttyS1 2022-03-10T03:00:33.442 DEBUG:teuthology.parallel:result is None 2022-03-10T03:00:33.442 INFO:teuthology.run_tasks:Running task internal.base... 2022-03-10T03:00:33.452 INFO:teuthology.task.internal:Creating test directory... 2022-03-10T03:00:33.453 DEBUG:teuthology.orchestra.run.smithi037:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2022-03-10T03:00:33.456 DEBUG:teuthology.orchestra.run.smithi038:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2022-03-10T03:00:33.459 DEBUG:teuthology.orchestra.run.smithi195:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2022-03-10T03:00:33.467 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2022-03-10T03:00:33.650 INFO:teuthology.run_tasks:Running task internal.archive... 2022-03-10T03:00:33.745 INFO:teuthology.task.internal:Creating archive directory... 2022-03-10T03:00:33.746 DEBUG:teuthology.orchestra.run.smithi037:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2022-03-10T03:00:33.749 DEBUG:teuthology.orchestra.run.smithi038:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2022-03-10T03:00:33.752 DEBUG:teuthology.orchestra.run.smithi195:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2022-03-10T03:00:33.780 INFO:teuthology.run_tasks:Running task internal.coredump... 2022-03-10T03:00:33.819 INFO:teuthology.task.internal:Enabling coredump saving... 2022-03-10T03:00:33.820 DEBUG:teuthology.orchestra.run.smithi037:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2022-03-10T03:00:33.823 DEBUG:teuthology.orchestra.run.smithi038:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2022-03-10T03:00:33.825 DEBUG:teuthology.orchestra.run.smithi195:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2022-03-10T03:00:33.842 INFO:teuthology.orchestra.run.smithi037.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-03-10T03:00:33.851 INFO:teuthology.orchestra.run.smithi037.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-03-10T03:00:33.867 INFO:teuthology.orchestra.run.smithi195.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-03-10T03:00:33.871 INFO:teuthology.orchestra.run.smithi038.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-03-10T03:00:33.877 INFO:teuthology.orchestra.run.smithi195.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-03-10T03:00:33.878 INFO:teuthology.orchestra.run.smithi038.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-03-10T03:00:33.881 INFO:teuthology.run_tasks:Running task internal.sudo... 2022-03-10T03:00:33.952 INFO:teuthology.task.internal:Configuring sudo... 2022-03-10T03:00:33.953 DEBUG:teuthology.orchestra.run.smithi037:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2022-03-10T03:00:33.955 DEBUG:teuthology.orchestra.run.smithi038:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2022-03-10T03:00:33.958 DEBUG:teuthology.orchestra.run.smithi195:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2022-03-10T03:00:33.976 INFO:teuthology.run_tasks:Running task internal.syslog... 2022-03-10T03:00:34.064 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2022-03-10T03:00:34.065 DEBUG:teuthology.orchestra.run.smithi037:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2022-03-10T03:00:34.068 DEBUG:teuthology.orchestra.run.smithi038:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2022-03-10T03:00:34.072 DEBUG:teuthology.orchestra.run.smithi195:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2022-03-10T03:00:34.080 DEBUG:teuthology.orchestra.run.smithi037:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2022-03-10T03:00:34.086 DEBUG:teuthology.orchestra.run.smithi037:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2022-03-10T03:00:34.137 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2022-03-10T03:00:34.138 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2022-03-10T03:00:34.155 DEBUG:teuthology.orchestra.run.smithi038:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2022-03-10T03:00:34.160 DEBUG:teuthology.orchestra.run.smithi038:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2022-03-10T03:00:34.211 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2022-03-10T03:00:34.211 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2022-03-10T03:00:34.296 DEBUG:teuthology.orchestra.run.smithi195:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2022-03-10T03:00:34.304 DEBUG:teuthology.orchestra.run.smithi195:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2022-03-10T03:00:34.312 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2022-03-10T03:00:34.313 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2022-03-10T03:00:34.360 DEBUG:teuthology.orchestra.run.smithi037:> sudo service rsyslog restart 2022-03-10T03:00:34.363 DEBUG:teuthology.orchestra.run.smithi038:> sudo service rsyslog restart 2022-03-10T03:00:34.365 DEBUG:teuthology.orchestra.run.smithi195:> sudo service rsyslog restart 2022-03-10T03:00:34.462 INFO:teuthology.run_tasks:Running task internal.timer... 2022-03-10T03:00:34.471 INFO:teuthology.task.internal:Starting timer... 2022-03-10T03:00:34.472 INFO:teuthology.run_tasks:Running task pcp... 2022-03-10T03:00:34.497 INFO:teuthology.run_tasks:Running task selinux... 2022-03-10T03:00:34.604 DEBUG:teuthology.task.selinux:Excluding smithi037: OS 'ubuntu' does not support SELinux 2022-03-10T03:00:34.605 DEBUG:teuthology.task.selinux:Excluding smithi038: OS 'ubuntu' does not support SELinux 2022-03-10T03:00:34.605 DEBUG:teuthology.task.selinux:Excluding smithi195: OS 'ubuntu' does not support SELinux 2022-03-10T03:00:34.605 DEBUG:teuthology.task.selinux:Getting current SELinux state 2022-03-10T03:00:34.606 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2022-03-10T03:00:34.606 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2022-03-10T03:00:34.606 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2022-03-10T03:00:34.794 DEBUG:teuthology.repo_utils:Setting repo remote to git://git.ceph.com/git/ceph-cm-ansible.git 2022-03-10T03:00:34.801 INFO:teuthology.repo_utils:Fetching master from origin 2022-03-10T03:00:34.838 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master to origin/master 2022-03-10T03:00:34.859 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2022-03-10T03:00:34.861 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi037.front.sepia.ceph.com,smithi038.front.sepia.ceph.com,smithi195.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/cephlab.yml 2022-03-10T03:03:40.791 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi037.front.sepia.ceph.com'), Remote(name='ubuntu@smithi038.front.sepia.ceph.com'), Remote(name='ubuntu@smithi195.front.sepia.ceph.com')] 2022-03-10T03:03:40.794 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2022-03-10T03:03:40.796 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi037.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-10T03:03:40.845 DEBUG:teuthology.orchestra.run.smithi037:> true 2022-03-10T03:03:40.969 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi038.front.sepia.ceph.com' 2022-03-10T03:03:40.969 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi038.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-10T03:03:41.019 DEBUG:teuthology.orchestra.run.smithi038:> true 2022-03-10T03:03:41.140 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi195.front.sepia.ceph.com' 2022-03-10T03:03:41.141 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi195.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-10T03:03:41.187 DEBUG:teuthology.orchestra.run.smithi195:> true 2022-03-10T03:03:41.317 INFO:teuthology.run_tasks:Running task clock... 2022-03-10T03:03:41.328 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2022-03-10T03:03:41.329 INFO:teuthology.orchestra.run:Running command with timeout 360 2022-03-10T03:03:41.331 DEBUG:teuthology.orchestra.run.smithi037:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-03-10T03:03:41.333 INFO:teuthology.orchestra.run:Running command with timeout 360 2022-03-10T03:03:41.334 DEBUG:teuthology.orchestra.run.smithi038:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-03-10T03:03:41.336 INFO:teuthology.orchestra.run:Running command with timeout 360 2022-03-10T03:03:41.336 DEBUG:teuthology.orchestra.run.smithi195:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-03-10T03:03:41.366 INFO:teuthology.orchestra.run.smithi037.stdout:10 Mar 03:03:41 ntpd[8961]: ntpd 4.2.8p12@1.3728-o (1): Starting 2022-03-10T03:03:41.366 INFO:teuthology.orchestra.run.smithi037.stdout:10 Mar 03:03:41 ntpd[8961]: Command line: ntpd -gq 2022-03-10T03:03:41.367 INFO:teuthology.orchestra.run.smithi037.stdout:10 Mar 03:03:41 ntpd[8961]: proto: precision = 0.060 usec (-24) 2022-03-10T03:03:41.367 INFO:teuthology.orchestra.run.smithi037.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-03-10T03:03:41.368 INFO:teuthology.orchestra.run.smithi037.stdout:10 Mar 03:03:41 ntpd[8961]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-03-10T03:03:41.368 INFO:teuthology.orchestra.run.smithi037.stderr:restrict ::: KOD does nothing without LIMITED. 2022-03-10T03:03:41.369 INFO:teuthology.orchestra.run.smithi037.stdout:10 Mar 03:03:41 ntpd[8961]: restrict ::: KOD does nothing without LIMITED. 2022-03-10T03:03:41.370 INFO:teuthology.orchestra.run.smithi037.stdout:10 Mar 03:03:41 ntpd[8961]: Listen and drop on 0 v6wildcard [::]:123 2022-03-10T03:03:41.370 INFO:teuthology.orchestra.run.smithi037.stdout:10 Mar 03:03:41 ntpd[8961]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2022-03-10T03:03:41.370 INFO:teuthology.orchestra.run.smithi037.stdout:10 Mar 03:03:41 ntpd[8961]: Listen normally on 2 lo 127.0.0.1:123 2022-03-10T03:03:41.371 INFO:teuthology.orchestra.run.smithi037.stdout:10 Mar 03:03:41 ntpd[8961]: Listen normally on 3 ens1f0 172.21.15.37:123 2022-03-10T03:03:41.371 INFO:teuthology.orchestra.run.smithi037.stdout:10 Mar 03:03:41 ntpd[8961]: Listen normally on 4 lo [::1]:123 2022-03-10T03:03:41.371 INFO:teuthology.orchestra.run.smithi037.stdout:10 Mar 03:03:41 ntpd[8961]: Listen normally on 5 ens1f0 [fe80::ec4:7aff:fe88:7794%4]:123 2022-03-10T03:03:41.371 INFO:teuthology.orchestra.run.smithi037.stdout:10 Mar 03:03:41 ntpd[8961]: Listening on routing socket on fd #22 for interface updates 2022-03-10T03:03:41.372 INFO:teuthology.orchestra.run.smithi038.stdout:10 Mar 03:03:42 ntpd[8926]: ntpd 4.2.8p12@1.3728-o (1): Starting 2022-03-10T03:03:41.372 INFO:teuthology.orchestra.run.smithi038.stdout:10 Mar 03:03:42 ntpd[8926]: Command line: ntpd -gq 2022-03-10T03:03:41.372 INFO:teuthology.orchestra.run.smithi038.stdout:10 Mar 03:03:42 ntpd[8926]: proto: precision = 0.080 usec (-23) 2022-03-10T03:03:41.373 INFO:teuthology.orchestra.run.smithi038.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-03-10T03:03:41.373 INFO:teuthology.orchestra.run.smithi038.stdout:10 Mar 03:03:42 ntpd[8926]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-03-10T03:03:41.374 INFO:teuthology.orchestra.run.smithi038.stderr:restrict ::: KOD does nothing without LIMITED. 2022-03-10T03:03:41.375 INFO:teuthology.orchestra.run.smithi038.stdout:10 Mar 03:03:42 ntpd[8926]: restrict ::: KOD does nothing without LIMITED. 2022-03-10T03:03:41.375 INFO:teuthology.orchestra.run.smithi038.stdout:10 Mar 03:03:42 ntpd[8926]: Listen and drop on 0 v6wildcard [::]:123 2022-03-10T03:03:41.375 INFO:teuthology.orchestra.run.smithi038.stdout:10 Mar 03:03:42 ntpd[8926]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2022-03-10T03:03:41.376 INFO:teuthology.orchestra.run.smithi038.stdout:10 Mar 03:03:42 ntpd[8926]: Listen normally on 2 lo 127.0.0.1:123 2022-03-10T03:03:41.376 INFO:teuthology.orchestra.run.smithi038.stdout:10 Mar 03:03:42 ntpd[8926]: Listen normally on 3 ens1f0 172.21.15.38:123 2022-03-10T03:03:41.376 INFO:teuthology.orchestra.run.smithi038.stdout:10 Mar 03:03:42 ntpd[8926]: Listen normally on 4 lo [::1]:123 2022-03-10T03:03:41.376 INFO:teuthology.orchestra.run.smithi038.stdout:10 Mar 03:03:42 ntpd[8926]: Listen normally on 5 ens1f0 [fe80::ec4:7aff:febd:15fc%3]:123 2022-03-10T03:03:41.377 INFO:teuthology.orchestra.run.smithi038.stdout:10 Mar 03:03:42 ntpd[8926]: Listening on routing socket on fd #22 for interface updates 2022-03-10T03:03:41.390 INFO:teuthology.orchestra.run.smithi195.stdout:10 Mar 03:03:41 ntpd[9031]: ntpd 4.2.8p12@1.3728-o (1): Starting 2022-03-10T03:03:41.390 INFO:teuthology.orchestra.run.smithi195.stdout:10 Mar 03:03:41 ntpd[9031]: Command line: ntpd -gq 2022-03-10T03:03:41.390 INFO:teuthology.orchestra.run.smithi195.stdout:10 Mar 03:03:41 ntpd[9031]: proto: precision = 0.100 usec (-23) 2022-03-10T03:03:41.391 INFO:teuthology.orchestra.run.smithi195.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-03-10T03:03:41.392 INFO:teuthology.orchestra.run.smithi195.stdout:10 Mar 03:03:41 ntpd[9031]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-03-10T03:03:41.392 INFO:teuthology.orchestra.run.smithi195.stderr:restrict ::: KOD does nothing without LIMITED. 2022-03-10T03:03:41.393 INFO:teuthology.orchestra.run.smithi195.stdout:10 Mar 03:03:41 ntpd[9031]: restrict ::: KOD does nothing without LIMITED. 2022-03-10T03:03:41.393 INFO:teuthology.orchestra.run.smithi195.stdout:10 Mar 03:03:41 ntpd[9031]: Listen and drop on 0 v6wildcard [::]:123 2022-03-10T03:03:41.393 INFO:teuthology.orchestra.run.smithi195.stdout:10 Mar 03:03:41 ntpd[9031]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2022-03-10T03:03:41.394 INFO:teuthology.orchestra.run.smithi195.stdout:10 Mar 03:03:41 ntpd[9031]: Listen normally on 2 lo 127.0.0.1:123 2022-03-10T03:03:41.394 INFO:teuthology.orchestra.run.smithi195.stdout:10 Mar 03:03:41 ntpd[9031]: Listen normally on 3 enp3s0f1 172.21.15.195:123 2022-03-10T03:03:41.394 INFO:teuthology.orchestra.run.smithi195.stdout:10 Mar 03:03:41 ntpd[9031]: Listen normally on 4 lo [::1]:123 2022-03-10T03:03:41.395 INFO:teuthology.orchestra.run.smithi195.stdout:10 Mar 03:03:41 ntpd[9031]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:8081%5]:123 2022-03-10T03:03:41.395 INFO:teuthology.orchestra.run.smithi195.stdout:10 Mar 03:03:41 ntpd[9031]: Listening on routing socket on fd #22 for interface updates 2022-03-10T03:03:42.367 INFO:teuthology.orchestra.run.smithi037.stderr:10 Mar 03:03:42 ntpd[8961]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T03:03:42.368 INFO:teuthology.orchestra.run.smithi037.stderr:10 Mar 03:03:42 ntpd[8961]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T03:03:42.368 INFO:teuthology.orchestra.run.smithi037.stderr:10 Mar 03:03:42 ntpd[8961]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T03:03:42.368 INFO:teuthology.orchestra.run.smithi037.stderr:10 Mar 03:03:42 ntpd[8961]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T03:03:42.369 INFO:teuthology.orchestra.run.smithi037.stderr:10 Mar 03:03:42 ntpd[8961]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T03:03:42.369 INFO:teuthology.orchestra.run.smithi037.stderr:10 Mar 03:03:42 ntpd[8961]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T03:03:42.369 INFO:teuthology.orchestra.run.smithi037.stderr:10 Mar 03:03:42 ntpd[8961]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-10T03:03:42.369 INFO:teuthology.orchestra.run.smithi037.stderr:10 Mar 03:03:42 ntpd[8961]: can't open /var/log/ntpstats/peerstats.20220310: Permission denied 2022-03-10T03:03:42.370 INFO:teuthology.orchestra.run.smithi037.stderr:10 Mar 03:03:42 ntpd[8961]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T03:03:42.370 INFO:teuthology.orchestra.run.smithi037.stderr:10 Mar 03:03:42 ntpd[8961]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T03:03:42.370 INFO:teuthology.orchestra.run.smithi037.stderr:10 Mar 03:03:42 ntpd[8961]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-10T03:03:42.370 INFO:teuthology.orchestra.run.smithi037.stderr:10 Mar 03:03:42 ntpd[8961]: can't open /var/log/ntpstats/peerstats.20220310: Permission denied 2022-03-10T03:03:42.372 INFO:teuthology.orchestra.run.smithi038.stderr:10 Mar 03:03:43 ntpd[8926]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T03:03:42.372 INFO:teuthology.orchestra.run.smithi038.stderr:10 Mar 03:03:43 ntpd[8926]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T03:03:42.372 INFO:teuthology.orchestra.run.smithi038.stderr:10 Mar 03:03:43 ntpd[8926]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T03:03:42.373 INFO:teuthology.orchestra.run.smithi038.stderr:10 Mar 03:03:43 ntpd[8926]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T03:03:42.373 INFO:teuthology.orchestra.run.smithi038.stderr:10 Mar 03:03:43 ntpd[8926]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T03:03:42.373 INFO:teuthology.orchestra.run.smithi038.stderr:10 Mar 03:03:43 ntpd[8926]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T03:03:42.373 INFO:teuthology.orchestra.run.smithi038.stderr:10 Mar 03:03:43 ntpd[8926]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-10T03:03:42.374 INFO:teuthology.orchestra.run.smithi038.stderr:10 Mar 03:03:43 ntpd[8926]: can't open /var/log/ntpstats/peerstats.20220310: Permission denied 2022-03-10T03:03:42.374 INFO:teuthology.orchestra.run.smithi038.stderr:10 Mar 03:03:43 ntpd[8926]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T03:03:42.374 INFO:teuthology.orchestra.run.smithi038.stderr:10 Mar 03:03:43 ntpd[8926]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T03:03:42.374 INFO:teuthology.orchestra.run.smithi038.stderr:10 Mar 03:03:43 ntpd[8926]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-10T03:03:42.375 INFO:teuthology.orchestra.run.smithi038.stderr:10 Mar 03:03:43 ntpd[8926]: can't open /var/log/ntpstats/peerstats.20220310: Permission denied 2022-03-10T03:03:42.390 INFO:teuthology.orchestra.run.smithi195.stderr:10 Mar 03:03:42 ntpd[9031]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T03:03:42.390 INFO:teuthology.orchestra.run.smithi195.stderr:10 Mar 03:03:42 ntpd[9031]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T03:03:42.391 INFO:teuthology.orchestra.run.smithi195.stderr:10 Mar 03:03:42 ntpd[9031]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T03:03:42.392 INFO:teuthology.orchestra.run.smithi195.stderr:10 Mar 03:03:42 ntpd[9031]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T03:03:42.392 INFO:teuthology.orchestra.run.smithi195.stderr:10 Mar 03:03:42 ntpd[9031]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-10T03:03:42.392 INFO:teuthology.orchestra.run.smithi195.stderr:10 Mar 03:03:42 ntpd[9031]: can't open /var/log/ntpstats/peerstats.20220310: Permission denied 2022-03-10T03:03:43.391 INFO:teuthology.orchestra.run.smithi195.stderr:10 Mar 03:03:43 ntpd[9031]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T03:03:43.392 INFO:teuthology.orchestra.run.smithi195.stderr:10 Mar 03:03:43 ntpd[9031]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T03:03:43.392 INFO:teuthology.orchestra.run.smithi195.stderr:10 Mar 03:03:43 ntpd[9031]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T03:03:43.392 INFO:teuthology.orchestra.run.smithi195.stderr:10 Mar 03:03:43 ntpd[9031]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T03:03:43.392 INFO:teuthology.orchestra.run.smithi195.stderr:10 Mar 03:03:43 ntpd[9031]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T03:03:43.393 INFO:teuthology.orchestra.run.smithi195.stderr:10 Mar 03:03:43 ntpd[9031]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T03:03:43.393 INFO:teuthology.orchestra.run.smithi195.stderr:10 Mar 03:03:43 ntpd[9031]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-10T03:03:43.393 INFO:teuthology.orchestra.run.smithi195.stderr:10 Mar 03:03:43 ntpd[9031]: can't open /var/log/ntpstats/peerstats.20220310: Permission denied 2022-03-10T03:03:43.393 INFO:teuthology.orchestra.run.smithi195.stderr:10 Mar 03:03:43 ntpd[9031]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T03:03:43.394 INFO:teuthology.orchestra.run.smithi195.stderr:10 Mar 03:03:43 ntpd[9031]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T03:03:43.394 INFO:teuthology.orchestra.run.smithi195.stderr:10 Mar 03:03:43 ntpd[9031]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-10T03:03:43.394 INFO:teuthology.orchestra.run.smithi195.stderr:10 Mar 03:03:43 ntpd[9031]: can't open /var/log/ntpstats/peerstats.20220310: Permission denied 2022-03-10T03:03:44.369 INFO:teuthology.orchestra.run.smithi037.stderr:10 Mar 03:03:44 ntpd[8961]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T03:03:44.370 INFO:teuthology.orchestra.run.smithi037.stderr:10 Mar 03:03:44 ntpd[8961]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T03:03:44.370 INFO:teuthology.orchestra.run.smithi037.stderr:10 Mar 03:03:44 ntpd[8961]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T03:03:44.370 INFO:teuthology.orchestra.run.smithi037.stderr:10 Mar 03:03:44 ntpd[8961]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T03:03:44.370 INFO:teuthology.orchestra.run.smithi037.stderr:10 Mar 03:03:44 ntpd[8961]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T03:03:44.371 INFO:teuthology.orchestra.run.smithi037.stderr:10 Mar 03:03:44 ntpd[8961]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T03:03:44.371 INFO:teuthology.orchestra.run.smithi037.stderr:10 Mar 03:03:44 ntpd[8961]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T03:03:44.371 INFO:teuthology.orchestra.run.smithi037.stderr:10 Mar 03:03:44 ntpd[8961]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T03:03:44.372 INFO:teuthology.orchestra.run.smithi037.stderr:10 Mar 03:03:44 ntpd[8961]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T03:03:44.372 INFO:teuthology.orchestra.run.smithi037.stderr:10 Mar 03:03:44 ntpd[8961]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T03:03:44.372 INFO:teuthology.orchestra.run.smithi037.stderr:10 Mar 03:03:44 ntpd[8961]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-10T03:03:44.372 INFO:teuthology.orchestra.run.smithi037.stderr:10 Mar 03:03:44 ntpd[8961]: can't open /var/log/ntpstats/peerstats.20220310: Permission denied 2022-03-10T03:03:44.373 INFO:teuthology.orchestra.run.smithi037.stderr:10 Mar 03:03:44 ntpd[8961]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T03:03:44.373 INFO:teuthology.orchestra.run.smithi037.stderr:10 Mar 03:03:44 ntpd[8961]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T03:03:44.373 INFO:teuthology.orchestra.run.smithi037.stderr:10 Mar 03:03:44 ntpd[8961]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-10T03:03:44.373 INFO:teuthology.orchestra.run.smithi037.stderr:10 Mar 03:03:44 ntpd[8961]: can't open /var/log/ntpstats/peerstats.20220310: Permission denied 2022-03-10T03:03:44.374 INFO:teuthology.orchestra.run.smithi037.stderr:10 Mar 03:03:44 ntpd[8961]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T03:03:44.374 INFO:teuthology.orchestra.run.smithi037.stderr:10 Mar 03:03:44 ntpd[8961]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T03:03:44.374 INFO:teuthology.orchestra.run.smithi037.stderr:10 Mar 03:03:44 ntpd[8961]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-10T03:03:44.375 INFO:teuthology.orchestra.run.smithi037.stderr:10 Mar 03:03:44 ntpd[8961]: can't open /var/log/ntpstats/peerstats.20220310: Permission denied 2022-03-10T03:03:44.375 INFO:teuthology.orchestra.run.smithi037.stderr:10 Mar 03:03:44 ntpd[8961]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T03:03:44.375 INFO:teuthology.orchestra.run.smithi037.stderr:10 Mar 03:03:44 ntpd[8961]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T03:03:44.375 INFO:teuthology.orchestra.run.smithi037.stderr:10 Mar 03:03:44 ntpd[8961]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-10T03:03:44.376 INFO:teuthology.orchestra.run.smithi037.stderr:10 Mar 03:03:44 ntpd[8961]: can't open /var/log/ntpstats/peerstats.20220310: Permission denied 2022-03-10T03:03:44.378 INFO:teuthology.orchestra.run.smithi038.stderr:10 Mar 03:03:45 ntpd[8926]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T03:03:44.378 INFO:teuthology.orchestra.run.smithi038.stderr:10 Mar 03:03:45 ntpd[8926]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T03:03:44.378 INFO:teuthology.orchestra.run.smithi038.stderr:10 Mar 03:03:45 ntpd[8926]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T03:03:44.379 INFO:teuthology.orchestra.run.smithi038.stderr:10 Mar 03:03:45 ntpd[8926]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T03:03:44.379 INFO:teuthology.orchestra.run.smithi038.stderr:10 Mar 03:03:45 ntpd[8926]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T03:03:44.379 INFO:teuthology.orchestra.run.smithi038.stderr:10 Mar 03:03:45 ntpd[8926]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T03:03:44.380 INFO:teuthology.orchestra.run.smithi038.stderr:10 Mar 03:03:45 ntpd[8926]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-10T03:03:44.380 INFO:teuthology.orchestra.run.smithi038.stderr:10 Mar 03:03:45 ntpd[8926]: can't open /var/log/ntpstats/peerstats.20220310: Permission denied 2022-03-10T03:03:44.380 INFO:teuthology.orchestra.run.smithi038.stderr:10 Mar 03:03:45 ntpd[8926]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T03:03:44.381 INFO:teuthology.orchestra.run.smithi038.stderr:10 Mar 03:03:45 ntpd[8926]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T03:03:44.381 INFO:teuthology.orchestra.run.smithi038.stderr:10 Mar 03:03:45 ntpd[8926]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-10T03:03:44.381 INFO:teuthology.orchestra.run.smithi038.stderr:10 Mar 03:03:45 ntpd[8926]: can't open /var/log/ntpstats/peerstats.20220310: Permission denied 2022-03-10T03:03:44.391 INFO:teuthology.orchestra.run.smithi195.stderr:10 Mar 03:03:44 ntpd[9031]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T03:03:44.392 INFO:teuthology.orchestra.run.smithi195.stderr:10 Mar 03:03:44 ntpd[9031]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T03:03:44.392 INFO:teuthology.orchestra.run.smithi195.stderr:10 Mar 03:03:44 ntpd[9031]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T03:03:44.393 INFO:teuthology.orchestra.run.smithi195.stderr:10 Mar 03:03:44 ntpd[9031]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T03:03:44.393 INFO:teuthology.orchestra.run.smithi195.stderr:10 Mar 03:03:44 ntpd[9031]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-10T03:03:44.393 INFO:teuthology.orchestra.run.smithi195.stderr:10 Mar 03:03:44 ntpd[9031]: can't open /var/log/ntpstats/peerstats.20220310: Permission denied 2022-03-10T03:03:45.392 INFO:teuthology.orchestra.run.smithi195.stderr:10 Mar 03:03:45 ntpd[9031]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T03:03:45.392 INFO:teuthology.orchestra.run.smithi195.stderr:10 Mar 03:03:45 ntpd[9031]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T03:03:45.392 INFO:teuthology.orchestra.run.smithi195.stderr:10 Mar 03:03:45 ntpd[9031]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T03:03:45.393 INFO:teuthology.orchestra.run.smithi195.stderr:10 Mar 03:03:45 ntpd[9031]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T03:03:45.393 INFO:teuthology.orchestra.run.smithi195.stderr:10 Mar 03:03:45 ntpd[9031]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T03:03:45.393 INFO:teuthology.orchestra.run.smithi195.stderr:10 Mar 03:03:45 ntpd[9031]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T03:03:45.393 INFO:teuthology.orchestra.run.smithi195.stderr:10 Mar 03:03:45 ntpd[9031]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-10T03:03:45.394 INFO:teuthology.orchestra.run.smithi195.stderr:10 Mar 03:03:45 ntpd[9031]: can't open /var/log/ntpstats/peerstats.20220310: Permission denied 2022-03-10T03:03:45.394 INFO:teuthology.orchestra.run.smithi195.stderr:10 Mar 03:03:45 ntpd[9031]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T03:03:45.394 INFO:teuthology.orchestra.run.smithi195.stderr:10 Mar 03:03:45 ntpd[9031]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T03:03:45.394 INFO:teuthology.orchestra.run.smithi195.stderr:10 Mar 03:03:45 ntpd[9031]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-10T03:03:45.394 INFO:teuthology.orchestra.run.smithi195.stderr:10 Mar 03:03:45 ntpd[9031]: can't open /var/log/ntpstats/peerstats.20220310: Permission denied 2022-03-10T03:03:46.368 INFO:teuthology.orchestra.run.smithi037.stderr:10 Mar 03:03:46 ntpd[8961]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T03:03:46.369 INFO:teuthology.orchestra.run.smithi037.stderr:10 Mar 03:03:46 ntpd[8961]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T03:03:46.369 INFO:teuthology.orchestra.run.smithi037.stderr:10 Mar 03:03:46 ntpd[8961]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T03:03:46.369 INFO:teuthology.orchestra.run.smithi037.stderr:10 Mar 03:03:46 ntpd[8961]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T03:03:46.370 INFO:teuthology.orchestra.run.smithi037.stderr:10 Mar 03:03:46 ntpd[8961]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T03:03:46.370 INFO:teuthology.orchestra.run.smithi037.stderr:10 Mar 03:03:46 ntpd[8961]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T03:03:46.370 INFO:teuthology.orchestra.run.smithi037.stderr:10 Mar 03:03:46 ntpd[8961]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T03:03:46.370 INFO:teuthology.orchestra.run.smithi037.stderr:10 Mar 03:03:46 ntpd[8961]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T03:03:46.370 INFO:teuthology.orchestra.run.smithi037.stderr:10 Mar 03:03:46 ntpd[8961]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T03:03:46.371 INFO:teuthology.orchestra.run.smithi037.stderr:10 Mar 03:03:46 ntpd[8961]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T03:03:46.371 INFO:teuthology.orchestra.run.smithi037.stderr:10 Mar 03:03:46 ntpd[8961]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-10T03:03:46.371 INFO:teuthology.orchestra.run.smithi037.stderr:10 Mar 03:03:46 ntpd[8961]: can't open /var/log/ntpstats/peerstats.20220310: Permission denied 2022-03-10T03:03:46.371 INFO:teuthology.orchestra.run.smithi037.stderr:10 Mar 03:03:46 ntpd[8961]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T03:03:46.372 INFO:teuthology.orchestra.run.smithi037.stderr:10 Mar 03:03:46 ntpd[8961]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T03:03:46.372 INFO:teuthology.orchestra.run.smithi037.stderr:10 Mar 03:03:46 ntpd[8961]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-10T03:03:46.372 INFO:teuthology.orchestra.run.smithi037.stderr:10 Mar 03:03:46 ntpd[8961]: can't open /var/log/ntpstats/peerstats.20220310: Permission denied 2022-03-10T03:03:46.372 INFO:teuthology.orchestra.run.smithi037.stderr:10 Mar 03:03:46 ntpd[8961]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T03:03:46.372 INFO:teuthology.orchestra.run.smithi037.stderr:10 Mar 03:03:46 ntpd[8961]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T03:03:46.373 INFO:teuthology.orchestra.run.smithi037.stderr:10 Mar 03:03:46 ntpd[8961]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-10T03:03:46.373 INFO:teuthology.orchestra.run.smithi037.stderr:10 Mar 03:03:46 ntpd[8961]: can't open /var/log/ntpstats/peerstats.20220310: Permission denied 2022-03-10T03:03:46.373 INFO:teuthology.orchestra.run.smithi037.stderr:10 Mar 03:03:46 ntpd[8961]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T03:03:46.373 INFO:teuthology.orchestra.run.smithi037.stderr:10 Mar 03:03:46 ntpd[8961]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T03:03:46.374 INFO:teuthology.orchestra.run.smithi037.stderr:10 Mar 03:03:46 ntpd[8961]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-10T03:03:46.374 INFO:teuthology.orchestra.run.smithi037.stderr:10 Mar 03:03:46 ntpd[8961]: can't open /var/log/ntpstats/peerstats.20220310: Permission denied 2022-03-10T03:03:46.377 INFO:teuthology.orchestra.run.smithi038.stderr:10 Mar 03:03:47 ntpd[8926]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T03:03:46.377 INFO:teuthology.orchestra.run.smithi038.stderr:10 Mar 03:03:47 ntpd[8926]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T03:03:46.378 INFO:teuthology.orchestra.run.smithi038.stderr:10 Mar 03:03:47 ntpd[8926]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T03:03:46.378 INFO:teuthology.orchestra.run.smithi038.stderr:10 Mar 03:03:47 ntpd[8926]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T03:03:46.378 INFO:teuthology.orchestra.run.smithi038.stderr:10 Mar 03:03:47 ntpd[8926]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T03:03:46.378 INFO:teuthology.orchestra.run.smithi038.stderr:10 Mar 03:03:47 ntpd[8926]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T03:03:46.379 INFO:teuthology.orchestra.run.smithi038.stderr:10 Mar 03:03:47 ntpd[8926]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T03:03:46.379 INFO:teuthology.orchestra.run.smithi038.stderr:10 Mar 03:03:47 ntpd[8926]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T03:03:46.379 INFO:teuthology.orchestra.run.smithi038.stderr:10 Mar 03:03:47 ntpd[8926]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T03:03:46.379 INFO:teuthology.orchestra.run.smithi038.stderr:10 Mar 03:03:47 ntpd[8926]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T03:03:46.380 INFO:teuthology.orchestra.run.smithi038.stderr:10 Mar 03:03:47 ntpd[8926]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-10T03:03:46.380 INFO:teuthology.orchestra.run.smithi038.stderr:10 Mar 03:03:47 ntpd[8926]: can't open /var/log/ntpstats/peerstats.20220310: Permission denied 2022-03-10T03:03:46.380 INFO:teuthology.orchestra.run.smithi038.stderr:10 Mar 03:03:47 ntpd[8926]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T03:03:46.380 INFO:teuthology.orchestra.run.smithi038.stderr:10 Mar 03:03:47 ntpd[8926]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T03:03:46.380 INFO:teuthology.orchestra.run.smithi038.stderr:10 Mar 03:03:47 ntpd[8926]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-10T03:03:46.381 INFO:teuthology.orchestra.run.smithi038.stderr:10 Mar 03:03:47 ntpd[8926]: can't open /var/log/ntpstats/peerstats.20220310: Permission denied 2022-03-10T03:03:46.381 INFO:teuthology.orchestra.run.smithi038.stderr:10 Mar 03:03:47 ntpd[8926]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T03:03:46.381 INFO:teuthology.orchestra.run.smithi038.stderr:10 Mar 03:03:47 ntpd[8926]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T03:03:46.381 INFO:teuthology.orchestra.run.smithi038.stderr:10 Mar 03:03:47 ntpd[8926]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-10T03:03:46.382 INFO:teuthology.orchestra.run.smithi038.stderr:10 Mar 03:03:47 ntpd[8926]: can't open /var/log/ntpstats/peerstats.20220310: Permission denied 2022-03-10T03:03:46.382 INFO:teuthology.orchestra.run.smithi038.stderr:10 Mar 03:03:47 ntpd[8926]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T03:03:46.382 INFO:teuthology.orchestra.run.smithi038.stderr:10 Mar 03:03:47 ntpd[8926]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T03:03:46.382 INFO:teuthology.orchestra.run.smithi038.stderr:10 Mar 03:03:47 ntpd[8926]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-10T03:03:46.382 INFO:teuthology.orchestra.run.smithi038.stderr:10 Mar 03:03:47 ntpd[8926]: can't open /var/log/ntpstats/peerstats.20220310: Permission denied 2022-03-10T03:03:46.391 INFO:teuthology.orchestra.run.smithi195.stderr:10 Mar 03:03:46 ntpd[9031]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T03:03:46.392 INFO:teuthology.orchestra.run.smithi195.stderr:10 Mar 03:03:46 ntpd[9031]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T03:03:46.392 INFO:teuthology.orchestra.run.smithi195.stderr:10 Mar 03:03:46 ntpd[9031]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T03:03:46.392 INFO:teuthology.orchestra.run.smithi195.stderr:10 Mar 03:03:46 ntpd[9031]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T03:03:46.392 INFO:teuthology.orchestra.run.smithi195.stderr:10 Mar 03:03:46 ntpd[9031]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T03:03:46.393 INFO:teuthology.orchestra.run.smithi195.stderr:10 Mar 03:03:46 ntpd[9031]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T03:03:46.393 INFO:teuthology.orchestra.run.smithi195.stderr:10 Mar 03:03:46 ntpd[9031]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-10T03:03:46.393 INFO:teuthology.orchestra.run.smithi195.stderr:10 Mar 03:03:46 ntpd[9031]: can't open /var/log/ntpstats/peerstats.20220310: Permission denied 2022-03-10T03:03:46.393 INFO:teuthology.orchestra.run.smithi195.stderr:10 Mar 03:03:46 ntpd[9031]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T03:03:46.394 INFO:teuthology.orchestra.run.smithi195.stderr:10 Mar 03:03:46 ntpd[9031]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T03:03:46.394 INFO:teuthology.orchestra.run.smithi195.stderr:10 Mar 03:03:46 ntpd[9031]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-10T03:03:46.394 INFO:teuthology.orchestra.run.smithi195.stderr:10 Mar 03:03:46 ntpd[9031]: can't open /var/log/ntpstats/peerstats.20220310: Permission denied 2022-03-10T03:03:47.391 INFO:teuthology.orchestra.run.smithi195.stderr:10 Mar 03:03:47 ntpd[9031]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T03:03:47.392 INFO:teuthology.orchestra.run.smithi195.stderr:10 Mar 03:03:47 ntpd[9031]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T03:03:47.392 INFO:teuthology.orchestra.run.smithi195.stderr:10 Mar 03:03:47 ntpd[9031]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T03:03:47.392 INFO:teuthology.orchestra.run.smithi195.stderr:10 Mar 03:03:47 ntpd[9031]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T03:03:47.392 INFO:teuthology.orchestra.run.smithi195.stderr:10 Mar 03:03:47 ntpd[9031]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T03:03:47.393 INFO:teuthology.orchestra.run.smithi195.stderr:10 Mar 03:03:47 ntpd[9031]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T03:03:47.393 INFO:teuthology.orchestra.run.smithi195.stderr:10 Mar 03:03:47 ntpd[9031]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-10T03:03:47.393 INFO:teuthology.orchestra.run.smithi195.stderr:10 Mar 03:03:47 ntpd[9031]: can't open /var/log/ntpstats/peerstats.20220310: Permission denied 2022-03-10T03:03:47.393 INFO:teuthology.orchestra.run.smithi195.stderr:10 Mar 03:03:47 ntpd[9031]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T03:03:47.394 INFO:teuthology.orchestra.run.smithi195.stderr:10 Mar 03:03:47 ntpd[9031]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T03:03:47.394 INFO:teuthology.orchestra.run.smithi195.stderr:10 Mar 03:03:47 ntpd[9031]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-10T03:03:47.394 INFO:teuthology.orchestra.run.smithi195.stderr:10 Mar 03:03:47 ntpd[9031]: can't open /var/log/ntpstats/peerstats.20220310: Permission denied 2022-03-10T03:03:48.368 INFO:teuthology.orchestra.run.smithi037.stderr:10 Mar 03:03:48 ntpd[8961]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T03:03:48.368 INFO:teuthology.orchestra.run.smithi037.stderr:10 Mar 03:03:48 ntpd[8961]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T03:03:48.368 INFO:teuthology.orchestra.run.smithi037.stderr:10 Mar 03:03:48 ntpd[8961]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T03:03:48.369 INFO:teuthology.orchestra.run.smithi037.stderr:10 Mar 03:03:48 ntpd[8961]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T03:03:48.369 INFO:teuthology.orchestra.run.smithi037.stderr:10 Mar 03:03:48 ntpd[8961]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T03:03:48.369 INFO:teuthology.orchestra.run.smithi037.stderr:10 Mar 03:03:48 ntpd[8961]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T03:03:48.369 INFO:teuthology.orchestra.run.smithi037.stderr:10 Mar 03:03:48 ntpd[8961]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T03:03:48.369 INFO:teuthology.orchestra.run.smithi037.stderr:10 Mar 03:03:48 ntpd[8961]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T03:03:48.370 INFO:teuthology.orchestra.run.smithi037.stderr:10 Mar 03:03:48 ntpd[8961]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T03:03:48.370 INFO:teuthology.orchestra.run.smithi037.stderr:10 Mar 03:03:48 ntpd[8961]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T03:03:48.370 INFO:teuthology.orchestra.run.smithi037.stderr:10 Mar 03:03:48 ntpd[8961]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-10T03:03:48.370 INFO:teuthology.orchestra.run.smithi037.stderr:10 Mar 03:03:48 ntpd[8961]: can't open /var/log/ntpstats/peerstats.20220310: Permission denied 2022-03-10T03:03:48.371 INFO:teuthology.orchestra.run.smithi037.stderr:10 Mar 03:03:48 ntpd[8961]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T03:03:48.371 INFO:teuthology.orchestra.run.smithi037.stderr:10 Mar 03:03:48 ntpd[8961]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T03:03:48.371 INFO:teuthology.orchestra.run.smithi037.stderr:10 Mar 03:03:48 ntpd[8961]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-10T03:03:48.371 INFO:teuthology.orchestra.run.smithi037.stderr:10 Mar 03:03:48 ntpd[8961]: can't open /var/log/ntpstats/peerstats.20220310: Permission denied 2022-03-10T03:03:48.372 INFO:teuthology.orchestra.run.smithi037.stdout:10 Mar 03:03:48 ntpd[8961]: ntpd: time slew +0.001360 s 2022-03-10T03:03:48.372 INFO:teuthology.orchestra.run.smithi037.stdout:ntpd: time slew +0.001360s 2022-03-10T03:03:48.373 INFO:teuthology.orchestra.run.smithi037.stderr:10 Mar 03:03:48 ntpd[8961]: can't open /var/log/ntpstats/loopstats.20220310: Permission denied 2022-03-10T03:03:48.375 INFO:teuthology.orchestra.run.smithi038.stderr:10 Mar 03:03:49 ntpd[8926]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T03:03:48.375 INFO:teuthology.orchestra.run.smithi038.stderr:10 Mar 03:03:49 ntpd[8926]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T03:03:48.375 INFO:teuthology.orchestra.run.smithi038.stderr:10 Mar 03:03:49 ntpd[8926]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T03:03:48.376 INFO:teuthology.orchestra.run.smithi038.stderr:10 Mar 03:03:49 ntpd[8926]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T03:03:48.376 INFO:teuthology.orchestra.run.smithi038.stderr:10 Mar 03:03:49 ntpd[8926]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T03:03:48.376 INFO:teuthology.orchestra.run.smithi038.stderr:10 Mar 03:03:49 ntpd[8926]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T03:03:48.376 INFO:teuthology.orchestra.run.smithi038.stderr:10 Mar 03:03:49 ntpd[8926]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T03:03:48.377 INFO:teuthology.orchestra.run.smithi038.stderr:10 Mar 03:03:49 ntpd[8926]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T03:03:48.377 INFO:teuthology.orchestra.run.smithi038.stderr:10 Mar 03:03:49 ntpd[8926]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T03:03:48.377 INFO:teuthology.orchestra.run.smithi038.stderr:10 Mar 03:03:49 ntpd[8926]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T03:03:48.378 INFO:teuthology.orchestra.run.smithi038.stderr:10 Mar 03:03:49 ntpd[8926]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-10T03:03:48.378 INFO:teuthology.orchestra.run.smithi038.stderr:10 Mar 03:03:49 ntpd[8926]: can't open /var/log/ntpstats/peerstats.20220310: Permission denied 2022-03-10T03:03:48.378 INFO:teuthology.orchestra.run.smithi038.stdout:10 Mar 03:03:48 ntpd[8926]: ntpd: time set -1.307593 s 2022-03-10T03:03:48.378 INFO:teuthology.orchestra.run.smithi038.stdout:ntpd: time set -1.307593s 2022-03-10T03:03:48.379 INFO:teuthology.orchestra.run.smithi038.stderr:10 Mar 03:03:48 ntpd[8926]: can't open /var/log/ntpstats/loopstats.20220310: Permission denied 2022-03-10T03:03:48.392 INFO:teuthology.orchestra.run.smithi195.stderr:10 Mar 03:03:48 ntpd[9031]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T03:03:48.393 INFO:teuthology.orchestra.run.smithi195.stderr:10 Mar 03:03:48 ntpd[9031]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T03:03:48.393 INFO:teuthology.orchestra.run.smithi195.stderr:10 Mar 03:03:48 ntpd[9031]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T03:03:48.393 INFO:teuthology.orchestra.run.smithi195.stderr:10 Mar 03:03:48 ntpd[9031]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T03:03:48.393 INFO:teuthology.orchestra.run.smithi195.stderr:10 Mar 03:03:48 ntpd[9031]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T03:03:48.394 INFO:teuthology.orchestra.run.smithi195.stderr:10 Mar 03:03:48 ntpd[9031]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T03:03:48.394 INFO:teuthology.orchestra.run.smithi195.stderr:10 Mar 03:03:48 ntpd[9031]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-10T03:03:48.394 INFO:teuthology.orchestra.run.smithi195.stderr:10 Mar 03:03:48 ntpd[9031]: can't open /var/log/ntpstats/peerstats.20220310: Permission denied 2022-03-10T03:03:48.394 INFO:teuthology.orchestra.run.smithi195.stderr:10 Mar 03:03:48 ntpd[9031]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T03:03:48.395 INFO:teuthology.orchestra.run.smithi195.stderr:10 Mar 03:03:48 ntpd[9031]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T03:03:48.395 INFO:teuthology.orchestra.run.smithi195.stderr:10 Mar 03:03:48 ntpd[9031]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-10T03:03:48.395 INFO:teuthology.orchestra.run.smithi195.stderr:10 Mar 03:03:48 ntpd[9031]: can't open /var/log/ntpstats/peerstats.20220310: Permission denied 2022-03-10T03:03:48.395 INFO:teuthology.orchestra.run.smithi195.stdout:10 Mar 03:03:48 ntpd[9031]: ntpd: time set +0.111575 s 2022-03-10T03:03:48.396 INFO:teuthology.orchestra.run.smithi195.stdout:ntpd: time set +0.111575s 2022-03-10T03:03:48.396 INFO:teuthology.orchestra.run.smithi195.stderr:10 Mar 03:03:48 ntpd[9031]: can't open /var/log/ntpstats/loopstats.20220310: Permission denied 2022-03-10T03:03:48.425 INFO:teuthology.orchestra.run.smithi037.stdout: remote refid st t when poll reach delay offset jitter 2022-03-10T03:03:48.425 INFO:teuthology.orchestra.run.smithi037.stdout:============================================================================== 2022-03-10T03:03:48.426 INFO:teuthology.orchestra.run.smithi037.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-10T03:03:48.426 INFO:teuthology.orchestra.run.smithi037.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-10T03:03:48.426 INFO:teuthology.orchestra.run.smithi037.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-10T03:03:48.426 INFO:teuthology.orchestra.run.smithi037.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-10T03:03:48.448 INFO:teuthology.orchestra.run.smithi195.stdout: remote refid st t when poll reach delay offset jitter 2022-03-10T03:03:48.449 INFO:teuthology.orchestra.run.smithi195.stdout:============================================================================== 2022-03-10T03:03:48.449 INFO:teuthology.orchestra.run.smithi195.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-10T03:03:48.449 INFO:teuthology.orchestra.run.smithi195.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-10T03:03:48.449 INFO:teuthology.orchestra.run.smithi195.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-10T03:03:48.450 INFO:teuthology.orchestra.run.smithi195.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-10T03:03:48.462 INFO:teuthology.orchestra.run.smithi038.stdout: remote refid st t when poll reach delay offset jitter 2022-03-10T03:03:48.463 INFO:teuthology.orchestra.run.smithi038.stdout:============================================================================== 2022-03-10T03:03:48.463 INFO:teuthology.orchestra.run.smithi038.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-10T03:03:48.463 INFO:teuthology.orchestra.run.smithi038.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-10T03:03:48.463 INFO:teuthology.orchestra.run.smithi038.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-10T03:03:48.464 INFO:teuthology.orchestra.run.smithi038.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-10T03:03:48.465 INFO:teuthology.run_tasks:Running task kubeadm... 2022-03-10T03:03:48.480 INFO:tasks.kubeadm:Kubeadm start 2022-03-10T03:03:48.481 INFO:tasks.kubeadm:Config: {'pod_network': 'calico', 'version': '1.21'} 2022-03-10T03:03:48.481 DEBUG:teuthology.orchestra.run.smithi037:> sudo modprobe br_netfilter && sudo sysctl net.bridge.bridge-nf-call-ip6tables=1 && sudo sysctl net.bridge.bridge-nf-call-iptables=1 && sudo sysctl net.ipv4.ip_forward=1 && sudo swapoff -a 2022-03-10T03:03:48.484 DEBUG:teuthology.orchestra.run.smithi038:> sudo modprobe br_netfilter && sudo sysctl net.bridge.bridge-nf-call-ip6tables=1 && sudo sysctl net.bridge.bridge-nf-call-iptables=1 && sudo sysctl net.ipv4.ip_forward=1 && sudo swapoff -a 2022-03-10T03:03:48.487 DEBUG:teuthology.orchestra.run.smithi195:> sudo modprobe br_netfilter && sudo sysctl net.bridge.bridge-nf-call-ip6tables=1 && sudo sysctl net.bridge.bridge-nf-call-iptables=1 && sudo sysctl net.ipv4.ip_forward=1 && sudo swapoff -a 2022-03-10T03:03:48.503 INFO:teuthology.orchestra.run.smithi038.stdout:net.bridge.bridge-nf-call-ip6tables = 1 2022-03-10T03:03:48.509 INFO:teuthology.orchestra.run.smithi037.stdout:net.bridge.bridge-nf-call-ip6tables = 1 2022-03-10T03:03:48.511 INFO:teuthology.orchestra.run.smithi038.stdout:net.bridge.bridge-nf-call-iptables = 1 2022-03-10T03:03:48.513 INFO:teuthology.orchestra.run.smithi195.stdout:net.bridge.bridge-nf-call-ip6tables = 1 2022-03-10T03:03:48.517 INFO:teuthology.orchestra.run.smithi037.stdout:net.bridge.bridge-nf-call-iptables = 1 2022-03-10T03:03:48.519 INFO:teuthology.orchestra.run.smithi038.stdout:net.ipv4.ip_forward = 1 2022-03-10T03:03:48.519 INFO:teuthology.orchestra.run.smithi195.stdout:net.bridge.bridge-nf-call-iptables = 1 2022-03-10T03:03:48.526 INFO:teuthology.orchestra.run.smithi037.stdout:net.ipv4.ip_forward = 1 2022-03-10T03:03:48.528 INFO:teuthology.orchestra.run.smithi195.stdout:net.ipv4.ip_forward = 1 2022-03-10T03:03:48.621 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2022-03-10T03:03:48.621 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd if=/etc/docker/daemon.json of=/dev/stdout 2022-03-10T03:03:48.638 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2022-03-10T03:03:48.639 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd of=/etc/docker/daemon.json 2022-03-10T03:03:48.698 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2022-03-10T03:03:48.699 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd if=/etc/docker/daemon.json of=/dev/stdout 2022-03-10T03:03:48.715 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2022-03-10T03:03:48.715 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/etc/docker/daemon.json 2022-03-10T03:03:48.730 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2022-03-10T03:03:48.731 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd if=/etc/docker/daemon.json of=/dev/stdout 2022-03-10T03:03:48.745 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2022-03-10T03:03:48.746 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd of=/etc/docker/daemon.json 2022-03-10T03:03:48.801 DEBUG:teuthology.orchestra.run.smithi037:> sudo systemctl restart docker || true 2022-03-10T03:03:48.804 DEBUG:teuthology.orchestra.run.smithi038:> sudo systemctl restart docker || true 2022-03-10T03:03:48.807 DEBUG:teuthology.orchestra.run.smithi195:> sudo systemctl restart docker || true 2022-03-10T03:03:49.732 INFO:tasks.kubeadm:Identifying pod subnet 2022-03-10T03:03:49.739 INFO:tasks.kubeadm:172.21.15.37 is in 172.21.0.0/20 at pos 3876 2022-03-10T03:03:49.751 INFO:tasks.kubeadm:Pod subnet: 10.249.32.0/21 2022-03-10T03:03:49.752 INFO:tasks.kubeadm:Installing kube{adm,ctl,let} 2022-03-10T03:03:49.752 DEBUG:teuthology.orchestra.run.smithi037:> sudo apt update && sudo apt install -y apt-transport-https ca-certificates curl && sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg && echo 'deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main' | sudo tee /etc/apt/sources.list.d/kubernetes.list && sudo apt update && sudo apt install -y kubelet kubeadm kubectl bridge-utils 2022-03-10T03:03:49.755 DEBUG:teuthology.orchestra.run.smithi038:> sudo apt update && sudo apt install -y apt-transport-https ca-certificates curl && sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg && echo 'deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main' | sudo tee /etc/apt/sources.list.d/kubernetes.list && sudo apt update && sudo apt install -y kubelet kubeadm kubectl bridge-utils 2022-03-10T03:03:49.758 DEBUG:teuthology.orchestra.run.smithi195:> sudo apt update && sudo apt install -y apt-transport-https ca-certificates curl && sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg && echo 'deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main' | sudo tee /etc/apt/sources.list.d/kubernetes.list && sudo apt update && sudo apt install -y kubelet kubeadm kubectl bridge-utils 2022-03-10T03:03:49.782 INFO:teuthology.orchestra.run.smithi038.stderr: 2022-03-10T03:03:49.782 INFO:teuthology.orchestra.run.smithi038.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-10T03:03:49.782 INFO:teuthology.orchestra.run.smithi038.stderr: 2022-03-10T03:03:49.783 INFO:teuthology.orchestra.run.smithi037.stderr: 2022-03-10T03:03:49.783 INFO:teuthology.orchestra.run.smithi037.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-10T03:03:49.783 INFO:teuthology.orchestra.run.smithi037.stderr: 2022-03-10T03:03:49.791 INFO:teuthology.orchestra.run.smithi195.stderr: 2022-03-10T03:03:49.792 INFO:teuthology.orchestra.run.smithi195.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-10T03:03:49.792 INFO:teuthology.orchestra.run.smithi195.stderr: 2022-03-10T03:03:49.899 INFO:teuthology.orchestra.run.smithi038.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-03-10T03:03:50.017 INFO:teuthology.orchestra.run.smithi038.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-10T03:03:50.027 INFO:teuthology.orchestra.run.smithi037.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-03-10T03:03:50.037 INFO:teuthology.orchestra.run.smithi195.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-10T03:03:50.038 INFO:teuthology.orchestra.run.smithi195.stdout:Hit:2 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-03-10T03:03:50.103 INFO:teuthology.orchestra.run.smithi038.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-03-10T03:03:50.109 INFO:teuthology.orchestra.run.smithi037.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-10T03:03:50.110 INFO:teuthology.orchestra.run.smithi195.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-03-10T03:03:50.186 INFO:teuthology.orchestra.run.smithi038.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-03-10T03:03:50.191 INFO:teuthology.orchestra.run.smithi037.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-03-10T03:03:50.193 INFO:teuthology.orchestra.run.smithi195.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-03-10T03:03:50.278 INFO:teuthology.orchestra.run.smithi037.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-03-10T03:03:51.412 INFO:teuthology.orchestra.run.smithi038.stdout:Reading package lists... 2022-03-10T03:03:51.510 INFO:teuthology.orchestra.run.smithi037.stdout:Reading package lists... 2022-03-10T03:03:51.517 INFO:teuthology.orchestra.run.smithi195.stdout:Reading package lists... 2022-03-10T03:03:51.588 INFO:teuthology.orchestra.run.smithi038.stdout:Building dependency tree... 2022-03-10T03:03:51.589 INFO:teuthology.orchestra.run.smithi038.stdout:Reading state information... 2022-03-10T03:03:51.681 INFO:teuthology.orchestra.run.smithi038.stdout:87 packages can be upgraded. Run 'apt list --upgradable' to see them. 2022-03-10T03:03:51.689 INFO:teuthology.orchestra.run.smithi037.stdout:Building dependency tree... 2022-03-10T03:03:51.689 INFO:teuthology.orchestra.run.smithi037.stdout:Reading state information... 2022-03-10T03:03:51.690 INFO:teuthology.orchestra.run.smithi195.stdout:Building dependency tree... 2022-03-10T03:03:51.690 INFO:teuthology.orchestra.run.smithi195.stdout:Reading state information... 2022-03-10T03:03:51.693 INFO:teuthology.orchestra.run.smithi038.stderr: 2022-03-10T03:03:51.693 INFO:teuthology.orchestra.run.smithi038.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-10T03:03:51.693 INFO:teuthology.orchestra.run.smithi038.stderr: 2022-03-10T03:03:51.747 INFO:teuthology.orchestra.run.smithi038.stdout:Reading package lists... 2022-03-10T03:03:51.780 INFO:teuthology.orchestra.run.smithi037.stdout:87 packages can be upgraded. Run 'apt list --upgradable' to see them. 2022-03-10T03:03:51.783 INFO:teuthology.orchestra.run.smithi195.stdout:87 packages can be upgraded. Run 'apt list --upgradable' to see them. 2022-03-10T03:03:51.795 INFO:teuthology.orchestra.run.smithi195.stderr: 2022-03-10T03:03:51.795 INFO:teuthology.orchestra.run.smithi195.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-10T03:03:51.796 INFO:teuthology.orchestra.run.smithi195.stderr: 2022-03-10T03:03:51.796 INFO:teuthology.orchestra.run.smithi037.stderr: 2022-03-10T03:03:51.796 INFO:teuthology.orchestra.run.smithi037.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-10T03:03:51.797 INFO:teuthology.orchestra.run.smithi037.stderr: 2022-03-10T03:03:51.848 INFO:teuthology.orchestra.run.smithi195.stdout:Reading package lists... 2022-03-10T03:03:51.852 INFO:teuthology.orchestra.run.smithi037.stdout:Reading package lists... 2022-03-10T03:03:51.909 INFO:teuthology.orchestra.run.smithi038.stdout:Building dependency tree... 2022-03-10T03:03:51.910 INFO:teuthology.orchestra.run.smithi038.stdout:Reading state information... 2022-03-10T03:03:52.008 INFO:teuthology.orchestra.run.smithi195.stdout:Building dependency tree... 2022-03-10T03:03:52.009 INFO:teuthology.orchestra.run.smithi195.stdout:Reading state information... 2022-03-10T03:03:52.030 INFO:teuthology.orchestra.run.smithi037.stdout:Building dependency tree... 2022-03-10T03:03:52.031 INFO:teuthology.orchestra.run.smithi037.stdout:Reading state information... 2022-03-10T03:03:52.083 INFO:teuthology.orchestra.run.smithi038.stdout:ca-certificates is already the newest version (20210119~20.04.2). 2022-03-10T03:03:52.083 INFO:teuthology.orchestra.run.smithi038.stdout:curl is already the newest version (7.68.0-1ubuntu2.7). 2022-03-10T03:03:52.083 INFO:teuthology.orchestra.run.smithi038.stdout:The following NEW packages will be installed: 2022-03-10T03:03:52.084 INFO:teuthology.orchestra.run.smithi038.stdout: apt-transport-https 2022-03-10T03:03:52.172 INFO:teuthology.orchestra.run.smithi195.stdout:ca-certificates is already the newest version (20210119~20.04.2). 2022-03-10T03:03:52.172 INFO:teuthology.orchestra.run.smithi195.stdout:curl is already the newest version (7.68.0-1ubuntu2.7). 2022-03-10T03:03:52.172 INFO:teuthology.orchestra.run.smithi195.stdout:The following NEW packages will be installed: 2022-03-10T03:03:52.176 INFO:teuthology.orchestra.run.smithi195.stdout: apt-transport-https 2022-03-10T03:03:52.208 INFO:teuthology.orchestra.run.smithi037.stdout:ca-certificates is already the newest version (20210119~20.04.2). 2022-03-10T03:03:52.209 INFO:teuthology.orchestra.run.smithi037.stdout:curl is already the newest version (7.68.0-1ubuntu2.7). 2022-03-10T03:03:52.209 INFO:teuthology.orchestra.run.smithi037.stdout:The following NEW packages will be installed: 2022-03-10T03:03:52.209 INFO:teuthology.orchestra.run.smithi037.stdout: apt-transport-https 2022-03-10T03:03:52.268 INFO:teuthology.orchestra.run.smithi038.stdout:0 upgraded, 1 newly installed, 0 to remove and 87 not upgraded. 2022-03-10T03:03:52.268 INFO:teuthology.orchestra.run.smithi038.stdout:Need to get 4,680 B of archives. 2022-03-10T03:03:52.268 INFO:teuthology.orchestra.run.smithi038.stdout:After this operation, 162 kB of additional disk space will be used. 2022-03-10T03:03:52.269 INFO:teuthology.orchestra.run.smithi038.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 apt-transport-https all 2.0.6 [4,680 B] 2022-03-10T03:03:52.356 INFO:teuthology.orchestra.run.smithi195.stdout:0 upgraded, 1 newly installed, 0 to remove and 87 not upgraded. 2022-03-10T03:03:52.357 INFO:teuthology.orchestra.run.smithi195.stdout:Need to get 4,680 B of archives. 2022-03-10T03:03:52.357 INFO:teuthology.orchestra.run.smithi195.stdout:After this operation, 162 kB of additional disk space will be used. 2022-03-10T03:03:52.357 INFO:teuthology.orchestra.run.smithi195.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 apt-transport-https all 2.0.6 [4,680 B] 2022-03-10T03:03:52.394 INFO:teuthology.orchestra.run.smithi037.stdout:0 upgraded, 1 newly installed, 0 to remove and 87 not upgraded. 2022-03-10T03:03:52.394 INFO:teuthology.orchestra.run.smithi037.stdout:Need to get 4,680 B of archives. 2022-03-10T03:03:52.394 INFO:teuthology.orchestra.run.smithi037.stdout:After this operation, 162 kB of additional disk space will be used. 2022-03-10T03:03:52.395 INFO:teuthology.orchestra.run.smithi037.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 apt-transport-https all 2.0.6 [4,680 B] 2022-03-10T03:03:52.496 INFO:teuthology.orchestra.run.smithi038.stderr:debconf: unable to initialize frontend: Dialog 2022-03-10T03:03:52.496 INFO:teuthology.orchestra.run.smithi038.stderr:debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) 2022-03-10T03:03:52.496 INFO:teuthology.orchestra.run.smithi038.stderr:debconf: falling back to frontend: Readline 2022-03-10T03:03:52.521 INFO:teuthology.orchestra.run.smithi038.stderr:debconf: unable to initialize frontend: Readline 2022-03-10T03:03:52.521 INFO:teuthology.orchestra.run.smithi038.stderr:debconf: (This frontend requires a controlling tty.) 2022-03-10T03:03:52.522 INFO:teuthology.orchestra.run.smithi038.stderr:debconf: falling back to frontend: Teletype 2022-03-10T03:03:52.527 INFO:teuthology.orchestra.run.smithi038.stderr:dpkg-preconfigure: unable to re-open stdin: 2022-03-10T03:03:52.577 INFO:teuthology.orchestra.run.smithi195.stderr:debconf: unable to initialize frontend: Dialog 2022-03-10T03:03:52.577 INFO:teuthology.orchestra.run.smithi195.stderr:debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) 2022-03-10T03:03:52.578 INFO:teuthology.orchestra.run.smithi195.stderr:debconf: falling back to frontend: Readline 2022-03-10T03:03:52.580 INFO:teuthology.orchestra.run.smithi038.stdout:Fetched 4,680 B in 0s (26.4 kB/s) 2022-03-10T03:03:52.593 INFO:teuthology.orchestra.run.smithi195.stderr:debconf: unable to initialize frontend: Readline 2022-03-10T03:03:52.593 INFO:teuthology.orchestra.run.smithi195.stderr:debconf: (This frontend requires a controlling tty.) 2022-03-10T03:03:52.594 INFO:teuthology.orchestra.run.smithi195.stderr:debconf: falling back to frontend: Teletype 2022-03-10T03:03:52.599 INFO:teuthology.orchestra.run.smithi195.stderr:dpkg-preconfigure: unable to re-open stdin: 2022-03-10T03:03:52.628 INFO:teuthology.orchestra.run.smithi037.stderr:debconf: unable to initialize frontend: Dialog 2022-03-10T03:03:52.628 INFO:teuthology.orchestra.run.smithi037.stderr:debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) 2022-03-10T03:03:52.628 INFO:teuthology.orchestra.run.smithi037.stderr:debconf: falling back to frontend: Readline 2022-03-10T03:03:52.645 INFO:teuthology.orchestra.run.smithi195.stdout:Fetched 4,680 B in 0s (26.5 kB/s) 2022-03-10T03:03:52.646 INFO:teuthology.orchestra.run.smithi037.stderr:debconf: unable to initialize frontend: Readline 2022-03-10T03:03:52.646 INFO:teuthology.orchestra.run.smithi037.stderr:debconf: (This frontend requires a controlling tty.) 2022-03-10T03:03:52.646 INFO:teuthology.orchestra.run.smithi037.stderr:debconf: falling back to frontend: Teletype 2022-03-10T03:03:52.652 INFO:teuthology.orchestra.run.smithi037.stderr:dpkg-preconfigure: unable to re-open stdin: 2022-03-10T03:03:52.667 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package apt-transport-https. 2022-03-10T03:03:52.705 INFO:teuthology.orchestra.run.smithi037.stdout:Fetched 4,680 B in 0s (26.4 kB/s) 2022-03-10T03:03:52.710 INFO:teuthology.orchestra.run.smithi038.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109240 files and directories currently installed.) 2022-03-10T03:03:52.713 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../apt-transport-https_2.0.6_all.deb ... 2022-03-10T03:03:52.737 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package apt-transport-https. 2022-03-10T03:03:52.757 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking apt-transport-https (2.0.6) ... 2022-03-10T03:03:52.778 INFO:teuthology.orchestra.run.smithi195.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109240 files and directories currently installed.) 2022-03-10T03:03:52.781 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../apt-transport-https_2.0.6_all.deb ... 2022-03-10T03:03:52.798 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package apt-transport-https. 2022-03-10T03:03:52.819 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking apt-transport-https (2.0.6) ... 2022-03-10T03:03:52.842 INFO:teuthology.orchestra.run.smithi037.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109240 files and directories currently installed.) 2022-03-10T03:03:52.846 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../apt-transport-https_2.0.6_all.deb ... 2022-03-10T03:03:52.888 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking apt-transport-https (2.0.6) ... 2022-03-10T03:03:53.002 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up apt-transport-https (2.0.6) ... 2022-03-10T03:03:53.065 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up apt-transport-https (2.0.6) ... 2022-03-10T03:03:53.159 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up apt-transport-https (2.0.6) ... 2022-03-10T03:03:53.582 INFO:teuthology.orchestra.run.smithi195.stdout:deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main 2022-03-10T03:03:53.594 INFO:teuthology.orchestra.run.smithi195.stderr: 2022-03-10T03:03:53.594 INFO:teuthology.orchestra.run.smithi195.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-10T03:03:53.594 INFO:teuthology.orchestra.run.smithi195.stderr: 2022-03-10T03:03:53.617 INFO:teuthology.orchestra.run.smithi038.stdout:deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main 2022-03-10T03:03:53.631 INFO:teuthology.orchestra.run.smithi038.stderr: 2022-03-10T03:03:53.632 INFO:teuthology.orchestra.run.smithi038.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-10T03:03:53.632 INFO:teuthology.orchestra.run.smithi038.stderr: 2022-03-10T03:03:53.677 INFO:teuthology.orchestra.run.smithi037.stdout:deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main 2022-03-10T03:03:53.690 INFO:teuthology.orchestra.run.smithi037.stderr: 2022-03-10T03:03:53.691 INFO:teuthology.orchestra.run.smithi037.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-10T03:03:53.691 INFO:teuthology.orchestra.run.smithi037.stderr: 2022-03-10T03:03:53.723 INFO:teuthology.orchestra.run.smithi195.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-03-10T03:03:53.765 INFO:teuthology.orchestra.run.smithi038.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-03-10T03:03:53.823 INFO:teuthology.orchestra.run.smithi037.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-03-10T03:03:53.833 INFO:teuthology.orchestra.run.smithi195.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-10T03:03:53.876 INFO:teuthology.orchestra.run.smithi038.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-10T03:03:53.915 INFO:teuthology.orchestra.run.smithi195.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-03-10T03:03:53.937 INFO:teuthology.orchestra.run.smithi037.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-10T03:03:53.958 INFO:teuthology.orchestra.run.smithi038.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-03-10T03:03:53.998 INFO:teuthology.orchestra.run.smithi195.stdout:Hit:5 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-03-10T03:03:54.017 INFO:teuthology.orchestra.run.smithi195.stdout:Get:2 https://packages.cloud.google.com/apt kubernetes-xenial InRelease [9,383 B] 2022-03-10T03:03:54.020 INFO:teuthology.orchestra.run.smithi037.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-03-10T03:03:54.045 INFO:teuthology.orchestra.run.smithi038.stdout:Hit:5 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-03-10T03:03:54.104 INFO:teuthology.orchestra.run.smithi037.stdout:Hit:5 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-03-10T03:03:54.208 INFO:teuthology.orchestra.run.smithi038.stdout:Get:2 https://packages.cloud.google.com/apt kubernetes-xenial InRelease [9,383 B] 2022-03-10T03:03:54.287 INFO:teuthology.orchestra.run.smithi037.stdout:Get:2 https://packages.cloud.google.com/apt kubernetes-xenial InRelease [9,383 B] 2022-03-10T03:03:54.360 INFO:teuthology.orchestra.run.smithi195.stdout:Get:6 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 Packages [53.9 kB] 2022-03-10T03:03:54.418 INFO:teuthology.orchestra.run.smithi038.stdout:Get:6 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 Packages [53.9 kB] 2022-03-10T03:03:54.499 INFO:teuthology.orchestra.run.smithi195.stdout:Fetched 63.3 kB in 1s (83.2 kB/s) 2022-03-10T03:03:54.512 INFO:teuthology.orchestra.run.smithi037.stdout:Get:6 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 Packages [53.9 kB] 2022-03-10T03:03:54.552 INFO:teuthology.orchestra.run.smithi038.stdout:Fetched 63.3 kB in 1s (81.9 kB/s) 2022-03-10T03:03:54.645 INFO:teuthology.orchestra.run.smithi037.stdout:Fetched 63.3 kB in 1s (78.7 kB/s) 2022-03-10T03:03:55.351 INFO:teuthology.orchestra.run.smithi195.stdout:Reading package lists... 2022-03-10T03:03:55.434 INFO:teuthology.orchestra.run.smithi038.stdout:Reading package lists... 2022-03-10T03:03:55.522 INFO:teuthology.orchestra.run.smithi037.stdout:Reading package lists... 2022-03-10T03:03:55.523 INFO:teuthology.orchestra.run.smithi195.stdout:Building dependency tree... 2022-03-10T03:03:55.523 INFO:teuthology.orchestra.run.smithi195.stdout:Reading state information... 2022-03-10T03:03:55.610 INFO:teuthology.orchestra.run.smithi038.stdout:Building dependency tree... 2022-03-10T03:03:55.611 INFO:teuthology.orchestra.run.smithi038.stdout:Reading state information... 2022-03-10T03:03:55.614 INFO:teuthology.orchestra.run.smithi195.stdout:87 packages can be upgraded. Run 'apt list --upgradable' to see them. 2022-03-10T03:03:55.630 INFO:teuthology.orchestra.run.smithi195.stderr: 2022-03-10T03:03:55.630 INFO:teuthology.orchestra.run.smithi195.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-10T03:03:55.631 INFO:teuthology.orchestra.run.smithi195.stderr: 2022-03-10T03:03:55.684 INFO:teuthology.orchestra.run.smithi195.stdout:Reading package lists... 2022-03-10T03:03:55.703 INFO:teuthology.orchestra.run.smithi037.stdout:Building dependency tree... 2022-03-10T03:03:55.704 INFO:teuthology.orchestra.run.smithi038.stdout:87 packages can be upgraded. Run 'apt list --upgradable' to see them. 2022-03-10T03:03:55.704 INFO:teuthology.orchestra.run.smithi037.stdout:Reading state information... 2022-03-10T03:03:55.719 INFO:teuthology.orchestra.run.smithi038.stderr: 2022-03-10T03:03:55.720 INFO:teuthology.orchestra.run.smithi038.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-10T03:03:55.720 INFO:teuthology.orchestra.run.smithi038.stderr: 2022-03-10T03:03:55.776 INFO:teuthology.orchestra.run.smithi038.stdout:Reading package lists... 2022-03-10T03:03:55.796 INFO:teuthology.orchestra.run.smithi037.stdout:87 packages can be upgraded. Run 'apt list --upgradable' to see them. 2022-03-10T03:03:55.813 INFO:teuthology.orchestra.run.smithi037.stderr: 2022-03-10T03:03:55.813 INFO:teuthology.orchestra.run.smithi037.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-10T03:03:55.813 INFO:teuthology.orchestra.run.smithi037.stderr: 2022-03-10T03:03:55.859 INFO:teuthology.orchestra.run.smithi195.stdout:Building dependency tree... 2022-03-10T03:03:55.860 INFO:teuthology.orchestra.run.smithi195.stdout:Reading state information... 2022-03-10T03:03:55.868 INFO:teuthology.orchestra.run.smithi037.stdout:Reading package lists... 2022-03-10T03:03:55.954 INFO:teuthology.orchestra.run.smithi038.stdout:Building dependency tree... 2022-03-10T03:03:55.954 INFO:teuthology.orchestra.run.smithi038.stdout:Reading state information... 2022-03-10T03:03:56.000 INFO:teuthology.orchestra.run.smithi195.stdout:bridge-utils is already the newest version (1.6-2ubuntu1). 2022-03-10T03:03:56.001 INFO:teuthology.orchestra.run.smithi195.stdout:bridge-utils set to manually installed. 2022-03-10T03:03:56.001 INFO:teuthology.orchestra.run.smithi195.stdout:The following additional packages will be installed: 2022-03-10T03:03:56.002 INFO:teuthology.orchestra.run.smithi195.stdout: conntrack cri-tools ebtables kubernetes-cni socat 2022-03-10T03:03:56.002 INFO:teuthology.orchestra.run.smithi195.stdout:Suggested packages: 2022-03-10T03:03:56.002 INFO:teuthology.orchestra.run.smithi195.stdout: nftables 2022-03-10T03:03:56.046 INFO:teuthology.orchestra.run.smithi037.stdout:Building dependency tree... 2022-03-10T03:03:56.046 INFO:teuthology.orchestra.run.smithi037.stdout:Reading state information... 2022-03-10T03:03:56.051 INFO:teuthology.orchestra.run.smithi195.stdout:The following NEW packages will be installed: 2022-03-10T03:03:56.052 INFO:teuthology.orchestra.run.smithi195.stdout: conntrack cri-tools ebtables kubeadm kubectl kubelet kubernetes-cni socat 2022-03-10T03:03:56.109 INFO:teuthology.orchestra.run.smithi038.stdout:bridge-utils is already the newest version (1.6-2ubuntu1). 2022-03-10T03:03:56.110 INFO:teuthology.orchestra.run.smithi038.stdout:bridge-utils set to manually installed. 2022-03-10T03:03:56.110 INFO:teuthology.orchestra.run.smithi038.stdout:The following additional packages will be installed: 2022-03-10T03:03:56.111 INFO:teuthology.orchestra.run.smithi038.stdout: conntrack cri-tools ebtables kubernetes-cni socat 2022-03-10T03:03:56.111 INFO:teuthology.orchestra.run.smithi038.stdout:Suggested packages: 2022-03-10T03:03:56.111 INFO:teuthology.orchestra.run.smithi038.stdout: nftables 2022-03-10T03:03:56.161 INFO:teuthology.orchestra.run.smithi038.stdout:The following NEW packages will be installed: 2022-03-10T03:03:56.162 INFO:teuthology.orchestra.run.smithi038.stdout: conntrack cri-tools ebtables kubeadm kubectl kubelet kubernetes-cni socat 2022-03-10T03:03:56.199 INFO:teuthology.orchestra.run.smithi037.stdout:bridge-utils is already the newest version (1.6-2ubuntu1). 2022-03-10T03:03:56.199 INFO:teuthology.orchestra.run.smithi037.stdout:bridge-utils set to manually installed. 2022-03-10T03:03:56.199 INFO:teuthology.orchestra.run.smithi037.stdout:The following additional packages will be installed: 2022-03-10T03:03:56.200 INFO:teuthology.orchestra.run.smithi037.stdout: conntrack cri-tools ebtables kubernetes-cni socat 2022-03-10T03:03:56.200 INFO:teuthology.orchestra.run.smithi037.stdout:Suggested packages: 2022-03-10T03:03:56.201 INFO:teuthology.orchestra.run.smithi037.stdout: nftables 2022-03-10T03:03:56.244 INFO:teuthology.orchestra.run.smithi195.stdout:0 upgraded, 8 newly installed, 0 to remove and 87 not upgraded. 2022-03-10T03:03:56.245 INFO:teuthology.orchestra.run.smithi195.stdout:Need to get 73.6 MB of archives. 2022-03-10T03:03:56.245 INFO:teuthology.orchestra.run.smithi195.stdout:After this operation, 321 MB of additional disk space will be used. 2022-03-10T03:03:56.245 INFO:teuthology.orchestra.run.smithi195.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 conntrack amd64 1:1.4.5-2 [30.3 kB] 2022-03-10T03:03:56.250 INFO:teuthology.orchestra.run.smithi037.stdout:The following NEW packages will be installed: 2022-03-10T03:03:56.251 INFO:teuthology.orchestra.run.smithi037.stdout: conntrack cri-tools ebtables kubeadm kubectl kubelet kubernetes-cni socat 2022-03-10T03:03:56.286 INFO:teuthology.orchestra.run.smithi195.stdout:Get:1 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 cri-tools amd64 1.19.0-00 [11.2 MB] 2022-03-10T03:03:56.363 INFO:teuthology.orchestra.run.smithi038.stdout:0 upgraded, 8 newly installed, 0 to remove and 87 not upgraded. 2022-03-10T03:03:56.363 INFO:teuthology.orchestra.run.smithi038.stdout:Need to get 73.6 MB of archives. 2022-03-10T03:03:56.364 INFO:teuthology.orchestra.run.smithi038.stdout:After this operation, 321 MB of additional disk space will be used. 2022-03-10T03:03:56.364 INFO:teuthology.orchestra.run.smithi038.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 conntrack amd64 1:1.4.5-2 [30.3 kB] 2022-03-10T03:03:56.398 INFO:teuthology.orchestra.run.smithi038.stdout:Get:1 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 cri-tools amd64 1.19.0-00 [11.2 MB] 2022-03-10T03:03:56.414 INFO:teuthology.orchestra.run.smithi195.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/main amd64 ebtables amd64 2.0.11-3build1 [80.3 kB] 2022-03-10T03:03:56.453 INFO:teuthology.orchestra.run.smithi037.stdout:0 upgraded, 8 newly installed, 0 to remove and 87 not upgraded. 2022-03-10T03:03:56.453 INFO:teuthology.orchestra.run.smithi037.stdout:Need to get 73.6 MB of archives. 2022-03-10T03:03:56.453 INFO:teuthology.orchestra.run.smithi037.stdout:After this operation, 321 MB of additional disk space will be used. 2022-03-10T03:03:56.453 INFO:teuthology.orchestra.run.smithi037.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 conntrack amd64 1:1.4.5-2 [30.3 kB] 2022-03-10T03:03:56.501 INFO:teuthology.orchestra.run.smithi037.stdout:Get:1 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 cri-tools amd64 1.19.0-00 [11.2 MB] 2022-03-10T03:03:56.534 INFO:teuthology.orchestra.run.smithi038.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/main amd64 ebtables amd64 2.0.11-3build1 [80.3 kB] 2022-03-10T03:03:56.535 INFO:teuthology.orchestra.run.smithi195.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2022-03-10T03:03:56.567 INFO:teuthology.orchestra.run.smithi195.stdout:Get:2 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubernetes-cni amd64 0.8.7-00 [25.0 MB] 2022-03-10T03:03:56.664 INFO:teuthology.orchestra.run.smithi037.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/main amd64 ebtables amd64 2.0.11-3build1 [80.3 kB] 2022-03-10T03:03:56.685 INFO:teuthology.orchestra.run.smithi038.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2022-03-10T03:03:56.754 INFO:teuthology.orchestra.run.smithi038.stdout:Get:2 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubernetes-cni amd64 0.8.7-00 [25.0 MB] 2022-03-10T03:03:56.784 INFO:teuthology.orchestra.run.smithi037.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2022-03-10T03:03:57.238 INFO:teuthology.orchestra.run.smithi195.stdout:Get:3 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubelet amd64 1.23.4-00 [19.5 MB] 2022-03-10T03:03:57.292 INFO:teuthology.orchestra.run.smithi038.stdout:Get:3 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubelet amd64 1.23.4-00 [19.5 MB] 2022-03-10T03:03:57.695 INFO:teuthology.orchestra.run.smithi038.stdout:Get:5 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubectl amd64 1.23.4-00 [8,927 kB] 2022-03-10T03:03:57.706 INFO:teuthology.orchestra.run.smithi195.stdout:Get:5 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubectl amd64 1.23.4-00 [8,927 kB] 2022-03-10T03:03:57.820 INFO:teuthology.orchestra.run.smithi038.stdout:Get:6 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubeadm amd64 1.23.4-00 [8,583 kB] 2022-03-10T03:03:57.858 INFO:teuthology.orchestra.run.smithi195.stdout:Get:6 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubeadm amd64 1.23.4-00 [8,583 kB] 2022-03-10T03:03:58.249 INFO:teuthology.orchestra.run.smithi037.stdout:Get:2 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubernetes-cni amd64 0.8.7-00 [25.0 MB] 2022-03-10T03:03:58.402 INFO:teuthology.orchestra.run.smithi038.stderr:debconf: unable to initialize frontend: Dialog 2022-03-10T03:03:58.402 INFO:teuthology.orchestra.run.smithi038.stderr:debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) 2022-03-10T03:03:58.402 INFO:teuthology.orchestra.run.smithi038.stderr:debconf: falling back to frontend: Readline 2022-03-10T03:03:58.411 INFO:teuthology.orchestra.run.smithi038.stderr:debconf: unable to initialize frontend: Readline 2022-03-10T03:03:58.411 INFO:teuthology.orchestra.run.smithi038.stderr:debconf: (This frontend requires a controlling tty.) 2022-03-10T03:03:58.411 INFO:teuthology.orchestra.run.smithi038.stderr:debconf: falling back to frontend: Teletype 2022-03-10T03:03:58.412 INFO:teuthology.orchestra.run.smithi195.stderr:debconf: unable to initialize frontend: Dialog 2022-03-10T03:03:58.412 INFO:teuthology.orchestra.run.smithi195.stderr:debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) 2022-03-10T03:03:58.412 INFO:teuthology.orchestra.run.smithi195.stderr:debconf: falling back to frontend: Readline 2022-03-10T03:03:58.416 INFO:teuthology.orchestra.run.smithi038.stderr:dpkg-preconfigure: unable to re-open stdin: 2022-03-10T03:03:58.420 INFO:teuthology.orchestra.run.smithi195.stderr:debconf: unable to initialize frontend: Readline 2022-03-10T03:03:58.420 INFO:teuthology.orchestra.run.smithi195.stderr:debconf: (This frontend requires a controlling tty.) 2022-03-10T03:03:58.421 INFO:teuthology.orchestra.run.smithi195.stderr:debconf: falling back to frontend: Teletype 2022-03-10T03:03:58.426 INFO:teuthology.orchestra.run.smithi195.stderr:dpkg-preconfigure: unable to re-open stdin: 2022-03-10T03:03:58.467 INFO:teuthology.orchestra.run.smithi038.stdout:Fetched 73.6 MB in 2s (36.7 MB/s) 2022-03-10T03:03:58.476 INFO:teuthology.orchestra.run.smithi195.stdout:Fetched 73.6 MB in 2s (34.6 MB/s) 2022-03-10T03:03:58.551 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package conntrack. 2022-03-10T03:03:58.561 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package conntrack. 2022-03-10T03:03:58.594 INFO:teuthology.orchestra.run.smithi038.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109244 files and directories currently installed.) 2022-03-10T03:03:58.598 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../0-conntrack_1%3a1.4.5-2_amd64.deb ... 2022-03-10T03:03:58.604 INFO:teuthology.orchestra.run.smithi195.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109244 files and directories currently installed.) 2022-03-10T03:03:58.607 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../0-conntrack_1%3a1.4.5-2_amd64.deb ... 2022-03-10T03:03:58.641 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking conntrack (1:1.4.5-2) ... 2022-03-10T03:03:58.651 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking conntrack (1:1.4.5-2) ... 2022-03-10T03:03:58.897 INFO:teuthology.orchestra.run.smithi037.stdout:Get:3 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubelet amd64 1.23.4-00 [19.5 MB] 2022-03-10T03:03:58.915 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package cri-tools. 2022-03-10T03:03:58.929 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../1-cri-tools_1.19.0-00_amd64.deb ... 2022-03-10T03:03:58.931 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package cri-tools. 2022-03-10T03:03:58.945 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../1-cri-tools_1.19.0-00_amd64.deb ... 2022-03-10T03:03:58.965 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking cri-tools (1.19.0-00) ... 2022-03-10T03:03:58.998 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking cri-tools (1.19.0-00) ... 2022-03-10T03:03:59.213 INFO:teuthology.orchestra.run.smithi037.stdout:Get:5 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubectl amd64 1.23.4-00 [8,927 kB] 2022-03-10T03:03:59.289 INFO:teuthology.orchestra.run.smithi037.stdout:Get:6 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubeadm amd64 1.23.4-00 [8,583 kB] 2022-03-10T03:03:59.641 INFO:teuthology.orchestra.run.smithi037.stderr:debconf: unable to initialize frontend: Dialog 2022-03-10T03:03:59.641 INFO:teuthology.orchestra.run.smithi037.stderr:debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) 2022-03-10T03:03:59.642 INFO:teuthology.orchestra.run.smithi037.stderr:debconf: falling back to frontend: Readline 2022-03-10T03:03:59.651 INFO:teuthology.orchestra.run.smithi037.stderr:debconf: unable to initialize frontend: Readline 2022-03-10T03:03:59.651 INFO:teuthology.orchestra.run.smithi037.stderr:debconf: (This frontend requires a controlling tty.) 2022-03-10T03:03:59.651 INFO:teuthology.orchestra.run.smithi037.stderr:debconf: falling back to frontend: Teletype 2022-03-10T03:03:59.657 INFO:teuthology.orchestra.run.smithi037.stderr:dpkg-preconfigure: unable to re-open stdin: 2022-03-10T03:03:59.707 INFO:teuthology.orchestra.run.smithi037.stdout:Fetched 73.6 MB in 3s (23.3 MB/s) 2022-03-10T03:03:59.807 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package conntrack. 2022-03-10T03:03:59.852 INFO:teuthology.orchestra.run.smithi037.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109244 files and directories currently installed.) 2022-03-10T03:03:59.855 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../0-conntrack_1%3a1.4.5-2_amd64.deb ... 2022-03-10T03:03:59.897 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking conntrack (1:1.4.5-2) ... 2022-03-10T03:04:00.061 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package ebtables. 2022-03-10T03:04:00.075 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../2-ebtables_2.0.11-3build1_amd64.deb ... 2022-03-10T03:04:00.085 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package ebtables. 2022-03-10T03:04:00.101 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../2-ebtables_2.0.11-3build1_amd64.deb ... 2022-03-10T03:04:00.135 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking ebtables (2.0.11-3build1) ... 2022-03-10T03:04:00.159 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking ebtables (2.0.11-3build1) ... 2022-03-10T03:04:00.185 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package cri-tools. 2022-03-10T03:04:00.199 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../1-cri-tools_1.19.0-00_amd64.deb ... 2022-03-10T03:04:00.235 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking cri-tools (1.19.0-00) ... 2022-03-10T03:04:00.400 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package kubernetes-cni. 2022-03-10T03:04:00.413 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package kubernetes-cni. 2022-03-10T03:04:00.413 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../3-kubernetes-cni_0.8.7-00_amd64.deb ... 2022-03-10T03:04:00.426 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../3-kubernetes-cni_0.8.7-00_amd64.deb ... 2022-03-10T03:04:00.450 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking kubernetes-cni (0.8.7-00) ... 2022-03-10T03:04:00.454 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking kubernetes-cni (0.8.7-00) ... 2022-03-10T03:04:01.399 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package ebtables. 2022-03-10T03:04:01.412 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../2-ebtables_2.0.11-3build1_amd64.deb ... 2022-03-10T03:04:01.473 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking ebtables (2.0.11-3build1) ... 2022-03-10T03:04:01.743 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package kubernetes-cni. 2022-03-10T03:04:01.756 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../3-kubernetes-cni_0.8.7-00_amd64.deb ... 2022-03-10T03:04:01.793 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking kubernetes-cni (0.8.7-00) ... 2022-03-10T03:04:01.869 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package socat. 2022-03-10T03:04:01.882 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../4-socat_1.7.3.3-2_amd64.deb ... 2022-03-10T03:04:01.890 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package socat. 2022-03-10T03:04:01.904 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../4-socat_1.7.3.3-2_amd64.deb ... 2022-03-10T03:04:01.919 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking socat (1.7.3.3-2) ... 2022-03-10T03:04:01.948 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking socat (1.7.3.3-2) ... 2022-03-10T03:04:02.314 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package kubelet. 2022-03-10T03:04:02.327 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package kubelet. 2022-03-10T03:04:02.328 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../5-kubelet_1.23.4-00_amd64.deb ... 2022-03-10T03:04:02.342 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../5-kubelet_1.23.4-00_amd64.deb ... 2022-03-10T03:04:02.372 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking kubelet (1.23.4-00) ... 2022-03-10T03:04:02.385 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking kubelet (1.23.4-00) ... 2022-03-10T03:04:03.237 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package socat. 2022-03-10T03:04:03.250 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../4-socat_1.7.3.3-2_amd64.deb ... 2022-03-10T03:04:03.287 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking socat (1.7.3.3-2) ... 2022-03-10T03:04:03.707 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package kubelet. 2022-03-10T03:04:03.722 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../5-kubelet_1.23.4-00_amd64.deb ... 2022-03-10T03:04:03.765 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking kubelet (1.23.4-00) ... 2022-03-10T03:04:04.750 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package kubectl. 2022-03-10T03:04:04.764 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../6-kubectl_1.23.4-00_amd64.deb ... 2022-03-10T03:04:04.808 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking kubectl (1.23.4-00) ... 2022-03-10T03:04:04.829 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package kubectl. 2022-03-10T03:04:04.842 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../6-kubectl_1.23.4-00_amd64.deb ... 2022-03-10T03:04:04.871 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking kubectl (1.23.4-00) ... 2022-03-10T03:04:05.952 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package kubeadm. 2022-03-10T03:04:05.967 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../7-kubeadm_1.23.4-00_amd64.deb ... 2022-03-10T03:04:06.010 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking kubeadm (1.23.4-00) ... 2022-03-10T03:04:06.032 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package kubeadm. 2022-03-10T03:04:06.047 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../7-kubeadm_1.23.4-00_amd64.deb ... 2022-03-10T03:04:06.082 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking kubeadm (1.23.4-00) ... 2022-03-10T03:04:06.168 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package kubectl. 2022-03-10T03:04:06.181 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../6-kubectl_1.23.4-00_amd64.deb ... 2022-03-10T03:04:06.218 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking kubectl (1.23.4-00) ... 2022-03-10T03:04:07.137 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up conntrack (1:1.4.5-2) ... 2022-03-10T03:04:07.201 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up conntrack (1:1.4.5-2) ... 2022-03-10T03:04:07.264 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up kubectl (1.23.4-00) ... 2022-03-10T03:04:07.328 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up kubectl (1.23.4-00) ... 2022-03-10T03:04:07.378 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package kubeadm. 2022-03-10T03:04:07.392 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../7-kubeadm_1.23.4-00_amd64.deb ... 2022-03-10T03:04:07.398 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up ebtables (2.0.11-3build1) ... 2022-03-10T03:04:07.437 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking kubeadm (1.23.4-00) ... 2022-03-10T03:04:07.453 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up ebtables (2.0.11-3build1) ... 2022-03-10T03:04:08.142 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up socat (1.7.3.3-2) ... 2022-03-10T03:04:08.256 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up socat (1.7.3.3-2) ... 2022-03-10T03:04:08.277 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up cri-tools (1.19.0-00) ... 2022-03-10T03:04:08.365 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up cri-tools (1.19.0-00) ... 2022-03-10T03:04:08.403 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up kubernetes-cni (0.8.7-00) ... 2022-03-10T03:04:08.491 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up kubernetes-cni (0.8.7-00) ... 2022-03-10T03:04:08.528 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up kubelet (1.23.4-00) ... 2022-03-10T03:04:08.553 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up conntrack (1:1.4.5-2) ... 2022-03-10T03:04:08.618 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up kubelet (1.23.4-00) ... 2022-03-10T03:04:08.684 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up kubectl (1.23.4-00) ... 2022-03-10T03:04:08.743 INFO:teuthology.orchestra.run.smithi195.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/kubelet.service -> /lib/systemd/system/kubelet.service. 2022-03-10T03:04:08.818 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up ebtables (2.0.11-3build1) ... 2022-03-10T03:04:08.845 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/kubelet.service -> /lib/systemd/system/kubelet.service. 2022-03-10T03:04:09.238 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up kubeadm (1.23.4-00) ... 2022-03-10T03:04:09.352 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up kubeadm (1.23.4-00) ... 2022-03-10T03:04:09.638 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up socat (1.7.3.3-2) ... 2022-03-10T03:04:09.763 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up cri-tools (1.19.0-00) ... 2022-03-10T03:04:09.774 INFO:teuthology.orchestra.run.smithi195.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-03-10T03:04:09.779 INFO:teuthology.orchestra.run.smithi038.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-03-10T03:04:09.873 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up kubernetes-cni (0.8.7-00) ... 2022-03-10T03:04:09.974 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up kubelet (1.23.4-00) ... 2022-03-10T03:04:10.207 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/kubelet.service -> /lib/systemd/system/kubelet.service. 2022-03-10T03:04:10.692 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up kubeadm (1.23.4-00) ... 2022-03-10T03:04:11.078 INFO:teuthology.orchestra.run.smithi037.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-03-10T03:04:11.946 DEBUG:teuthology.orchestra.run.smithi037:> sudo systemctl enable --now kubelet && sudo kubeadm config images pull 2022-03-10T03:04:11.991 DEBUG:teuthology.orchestra.run.smithi038:> sudo systemctl enable --now kubelet && sudo kubeadm config images pull 2022-03-10T03:04:11.994 DEBUG:teuthology.orchestra.run.smithi195:> sudo systemctl enable --now kubelet && sudo kubeadm config images pull 2022-03-10T03:04:24.252 INFO:teuthology.orchestra.run.smithi195.stdout:[config/images] Pulled k8s.gcr.io/kube-apiserver:v1.23.4 2022-03-10T03:04:24.891 INFO:teuthology.orchestra.run.smithi038.stdout:[config/images] Pulled k8s.gcr.io/kube-apiserver:v1.23.4 2022-03-10T03:04:25.810 INFO:teuthology.orchestra.run.smithi037.stdout:[config/images] Pulled k8s.gcr.io/kube-apiserver:v1.23.4 2022-03-10T03:04:27.154 INFO:teuthology.orchestra.run.smithi195.stdout:[config/images] Pulled k8s.gcr.io/kube-controller-manager:v1.23.4 2022-03-10T03:04:28.260 INFO:teuthology.orchestra.run.smithi038.stdout:[config/images] Pulled k8s.gcr.io/kube-controller-manager:v1.23.4 2022-03-10T03:04:29.004 INFO:teuthology.orchestra.run.smithi037.stdout:[config/images] Pulled k8s.gcr.io/kube-controller-manager:v1.23.4 2022-03-10T03:04:29.527 INFO:teuthology.orchestra.run.smithi195.stdout:[config/images] Pulled k8s.gcr.io/kube-scheduler:v1.23.4 2022-03-10T03:04:30.973 INFO:teuthology.orchestra.run.smithi038.stdout:[config/images] Pulled k8s.gcr.io/kube-scheduler:v1.23.4 2022-03-10T03:04:31.676 INFO:teuthology.orchestra.run.smithi037.stdout:[config/images] Pulled k8s.gcr.io/kube-scheduler:v1.23.4 2022-03-10T03:04:33.693 INFO:teuthology.orchestra.run.smithi195.stdout:[config/images] Pulled k8s.gcr.io/kube-proxy:v1.23.4 2022-03-10T03:04:35.299 INFO:teuthology.orchestra.run.smithi195.stdout:[config/images] Pulled k8s.gcr.io/pause:3.6 2022-03-10T03:04:35.373 INFO:teuthology.orchestra.run.smithi038.stdout:[config/images] Pulled k8s.gcr.io/kube-proxy:v1.23.4 2022-03-10T03:04:35.751 INFO:teuthology.orchestra.run.smithi037.stdout:[config/images] Pulled k8s.gcr.io/kube-proxy:v1.23.4 2022-03-10T03:04:37.407 INFO:teuthology.orchestra.run.smithi038.stdout:[config/images] Pulled k8s.gcr.io/pause:3.6 2022-03-10T03:04:37.533 INFO:teuthology.orchestra.run.smithi037.stdout:[config/images] Pulled k8s.gcr.io/pause:3.6 2022-03-10T03:04:41.017 INFO:teuthology.orchestra.run.smithi195.stdout:[config/images] Pulled k8s.gcr.io/etcd:3.5.1-0 2022-03-10T03:04:43.533 INFO:teuthology.orchestra.run.smithi195.stdout:[config/images] Pulled k8s.gcr.io/coredns/coredns:v1.8.6 2022-03-10T03:04:44.024 INFO:teuthology.orchestra.run.smithi037.stdout:[config/images] Pulled k8s.gcr.io/etcd:3.5.1-0 2022-03-10T03:04:45.347 INFO:teuthology.orchestra.run.smithi038.stdout:[config/images] Pulled k8s.gcr.io/etcd:3.5.1-0 2022-03-10T03:04:46.357 INFO:teuthology.orchestra.run.smithi037.stdout:[config/images] Pulled k8s.gcr.io/coredns/coredns:v1.8.6 2022-03-10T03:04:47.505 INFO:teuthology.orchestra.run.smithi038.stdout:[config/images] Pulled k8s.gcr.io/coredns/coredns:v1.8.6 2022-03-10T03:04:47.509 INFO:tasks.kubeadm:Token: abcdef.20k7qcj4lgeb4yo9 2022-03-10T03:04:47.509 INFO:tasks.kubeadm:Remotes: {Remote(name='ubuntu@smithi037.front.sepia.ceph.com'): '172.21.15.37', Remote(name='ubuntu@smithi038.front.sepia.ceph.com'): '172.21.15.38', Remote(name='ubuntu@smithi195.front.sepia.ceph.com'): '172.21.15.195'} 2022-03-10T03:04:47.510 DEBUG:teuthology.orchestra.run.smithi037:> sudo kubeadm init --node-name smithi037 --token abcdef.20k7qcj4lgeb4yo9 --pod-network-cidr 10.249.32.0/21 2022-03-10T03:04:47.743 INFO:teuthology.orchestra.run.smithi037.stdout:[init] Using Kubernetes version: v1.23.4 2022-03-10T03:04:47.744 INFO:teuthology.orchestra.run.smithi037.stdout:[preflight] Running pre-flight checks 2022-03-10T03:04:48.166 INFO:teuthology.orchestra.run.smithi037.stdout:[preflight] Pulling images required for setting up a Kubernetes cluster 2022-03-10T03:04:48.166 INFO:teuthology.orchestra.run.smithi037.stdout:[preflight] This might take a minute or two, depending on the speed of your internet connection 2022-03-10T03:04:48.167 INFO:teuthology.orchestra.run.smithi037.stdout:[preflight] You can also perform this action in beforehand using 'kubeadm config images pull' 2022-03-10T03:04:48.488 INFO:teuthology.orchestra.run.smithi037.stdout:[certs] Using certificateDir folder "/etc/kubernetes/pki" 2022-03-10T03:04:48.607 INFO:teuthology.orchestra.run.smithi037.stdout:[certs] Generating "ca" certificate and key 2022-03-10T03:04:48.751 INFO:teuthology.orchestra.run.smithi037.stdout:[certs] Generating "apiserver" certificate and key 2022-03-10T03:04:48.752 INFO:teuthology.orchestra.run.smithi037.stdout:[certs] apiserver serving cert is signed for DNS names [kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local smithi037] and IPs [10.96.0.1 172.21.15.37] 2022-03-10T03:04:48.974 INFO:teuthology.orchestra.run.smithi037.stdout:[certs] Generating "apiserver-kubelet-client" certificate and key 2022-03-10T03:04:49.150 INFO:teuthology.orchestra.run.smithi037.stdout:[certs] Generating "front-proxy-ca" certificate and key 2022-03-10T03:04:49.371 INFO:teuthology.orchestra.run.smithi037.stdout:[certs] Generating "front-proxy-client" certificate and key 2022-03-10T03:04:49.506 INFO:teuthology.orchestra.run.smithi037.stdout:[certs] Generating "etcd/ca" certificate and key 2022-03-10T03:04:49.685 INFO:teuthology.orchestra.run.smithi037.stdout:[certs] Generating "etcd/server" certificate and key 2022-03-10T03:04:49.685 INFO:teuthology.orchestra.run.smithi037.stdout:[certs] etcd/server serving cert is signed for DNS names [localhost smithi037] and IPs [172.21.15.37 127.0.0.1 ::1] 2022-03-10T03:04:49.786 INFO:teuthology.orchestra.run.smithi037.stdout:[certs] Generating "etcd/peer" certificate and key 2022-03-10T03:04:49.787 INFO:teuthology.orchestra.run.smithi037.stdout:[certs] etcd/peer serving cert is signed for DNS names [localhost smithi037] and IPs [172.21.15.37 127.0.0.1 ::1] 2022-03-10T03:04:49.850 INFO:teuthology.orchestra.run.smithi037.stdout:[certs] Generating "etcd/healthcheck-client" certificate and key 2022-03-10T03:04:49.899 INFO:teuthology.orchestra.run.smithi037.stdout:[certs] Generating "apiserver-etcd-client" certificate and key 2022-03-10T03:04:50.214 INFO:teuthology.orchestra.run.smithi037.stdout:[certs] Generating "sa" key and public key 2022-03-10T03:04:50.214 INFO:teuthology.orchestra.run.smithi037.stdout:[kubeconfig] Using kubeconfig folder "/etc/kubernetes" 2022-03-10T03:04:50.575 INFO:teuthology.orchestra.run.smithi037.stdout:[kubeconfig] Writing "admin.conf" kubeconfig file 2022-03-10T03:04:50.763 INFO:teuthology.orchestra.run.smithi037.stdout:[kubeconfig] Writing "kubelet.conf" kubeconfig file 2022-03-10T03:04:50.972 INFO:teuthology.orchestra.run.smithi037.stdout:[kubeconfig] Writing "controller-manager.conf" kubeconfig file 2022-03-10T03:04:51.270 INFO:teuthology.orchestra.run.smithi037.stdout:[kubeconfig] Writing "scheduler.conf" kubeconfig file 2022-03-10T03:04:51.286 INFO:teuthology.orchestra.run.smithi037.stdout:[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" 2022-03-10T03:04:51.301 INFO:teuthology.orchestra.run.smithi037.stdout:[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" 2022-03-10T03:04:51.301 INFO:teuthology.orchestra.run.smithi037.stdout:[kubelet-start] Starting the kubelet 2022-03-10T03:04:51.511 INFO:teuthology.orchestra.run.smithi037.stdout:[control-plane] Using manifest folder "/etc/kubernetes/manifests" 2022-03-10T03:04:51.512 INFO:teuthology.orchestra.run.smithi037.stdout:[control-plane] Creating static Pod manifest for "kube-apiserver" 2022-03-10T03:04:51.514 INFO:teuthology.orchestra.run.smithi037.stdout:[control-plane] Creating static Pod manifest for "kube-controller-manager" 2022-03-10T03:04:51.515 INFO:teuthology.orchestra.run.smithi037.stdout:[control-plane] Creating static Pod manifest for "kube-scheduler" 2022-03-10T03:04:51.515 INFO:teuthology.orchestra.run.smithi037.stdout:[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests" 2022-03-10T03:04:51.518 INFO:teuthology.orchestra.run.smithi037.stdout:[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s 2022-03-10T03:05:08.534 INFO:teuthology.orchestra.run.smithi037.stdout:[apiclient] All control plane components are healthy after 17.015031 seconds 2022-03-10T03:05:08.534 INFO:teuthology.orchestra.run.smithi037.stdout:[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace 2022-03-10T03:05:08.554 INFO:teuthology.orchestra.run.smithi037.stdout:[kubelet] Creating a ConfigMap "kubelet-config-1.23" in namespace kube-system with the configuration for the kubelets in the cluster 2022-03-10T03:05:08.555 INFO:teuthology.orchestra.run.smithi037.stdout:NOTE: The "kubelet-config-1.23" naming of the kubelet ConfigMap is deprecated. Once the UnversionedKubeletConfigMap feature gate graduates to Beta the default name will become just "kubelet-config". Kubeadm upgrade will handle this transition transparently. 2022-03-10T03:05:09.107 INFO:teuthology.orchestra.run.smithi037.stdout:[upload-certs] Skipping phase. Please see --upload-certs 2022-03-10T03:05:09.108 INFO:teuthology.orchestra.run.smithi037.stdout:[mark-control-plane] Marking the node smithi037 as control-plane by adding the labels: [node-role.kubernetes.io/master(deprecated) node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers] 2022-03-10T03:05:09.108 INFO:teuthology.orchestra.run.smithi037.stdout:[mark-control-plane] Marking the node smithi037 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule] 2022-03-10T03:05:09.619 INFO:teuthology.orchestra.run.smithi037.stdout:[bootstrap-token] Using token: abcdef.20k7qcj4lgeb4yo9 2022-03-10T03:05:09.619 INFO:teuthology.orchestra.run.smithi037.stdout:[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles 2022-03-10T03:05:09.654 INFO:teuthology.orchestra.run.smithi037.stdout:[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes 2022-03-10T03:05:09.671 INFO:teuthology.orchestra.run.smithi037.stdout:[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials 2022-03-10T03:05:09.734 INFO:teuthology.orchestra.run.smithi037.stdout:[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token 2022-03-10T03:05:09.737 INFO:teuthology.orchestra.run.smithi037.stdout:[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster 2022-03-10T03:05:09.746 INFO:teuthology.orchestra.run.smithi037.stdout:[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace 2022-03-10T03:05:09.788 INFO:teuthology.orchestra.run.smithi037.stdout:[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key 2022-03-10T03:05:10.265 INFO:teuthology.orchestra.run.smithi037.stdout:[addons] Applied essential addon: CoreDNS 2022-03-10T03:05:10.346 INFO:teuthology.orchestra.run.smithi037.stdout:[addons] Applied essential addon: kube-proxy 2022-03-10T03:05:10.347 INFO:teuthology.orchestra.run.smithi037.stdout: 2022-03-10T03:05:10.347 INFO:teuthology.orchestra.run.smithi037.stdout:Your Kubernetes control-plane has initialized successfully! 2022-03-10T03:05:10.348 INFO:teuthology.orchestra.run.smithi037.stdout: 2022-03-10T03:05:10.348 INFO:teuthology.orchestra.run.smithi037.stdout:To start using your cluster, you need to run the following as a regular user: 2022-03-10T03:05:10.348 INFO:teuthology.orchestra.run.smithi037.stdout: 2022-03-10T03:05:10.348 INFO:teuthology.orchestra.run.smithi037.stdout: mkdir -p $HOME/.kube 2022-03-10T03:05:10.349 INFO:teuthology.orchestra.run.smithi037.stdout: sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config 2022-03-10T03:05:10.349 INFO:teuthology.orchestra.run.smithi037.stdout: sudo chown $(id -u):$(id -g) $HOME/.kube/config 2022-03-10T03:05:10.349 INFO:teuthology.orchestra.run.smithi037.stdout: 2022-03-10T03:05:10.349 INFO:teuthology.orchestra.run.smithi037.stdout:Alternatively, if you are the root user, you can run: 2022-03-10T03:05:10.349 INFO:teuthology.orchestra.run.smithi037.stdout: 2022-03-10T03:05:10.350 INFO:teuthology.orchestra.run.smithi037.stdout: export KUBECONFIG=/etc/kubernetes/admin.conf 2022-03-10T03:05:10.350 INFO:teuthology.orchestra.run.smithi037.stdout: 2022-03-10T03:05:10.350 INFO:teuthology.orchestra.run.smithi037.stdout:You should now deploy a pod network to the cluster. 2022-03-10T03:05:10.350 INFO:teuthology.orchestra.run.smithi037.stdout:Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at: 2022-03-10T03:05:10.351 INFO:teuthology.orchestra.run.smithi037.stdout: https://kubernetes.io/docs/concepts/cluster-administration/addons/ 2022-03-10T03:05:10.351 INFO:teuthology.orchestra.run.smithi037.stdout: 2022-03-10T03:05:10.351 INFO:teuthology.orchestra.run.smithi037.stdout:Then you can join any number of worker nodes by running the following on each as root: 2022-03-10T03:05:10.351 INFO:teuthology.orchestra.run.smithi037.stdout: 2022-03-10T03:05:10.351 INFO:teuthology.orchestra.run.smithi037.stdout:kubeadm join 172.21.15.37:6443 --token abcdef.20k7qcj4lgeb4yo9 \ 2022-03-10T03:05:10.352 INFO:teuthology.orchestra.run.smithi037.stdout: --discovery-token-ca-cert-hash sha256:41d0e3922c23de3561d48e88e7f1317ce359f39843c992ac7924a32592de2a1a 2022-03-10T03:05:10.353 DEBUG:teuthology.orchestra.run.smithi038:> sudo kubeadm join 172.21.15.37:6443 --node-name smithi038 --token abcdef.20k7qcj4lgeb4yo9 --discovery-token-unsafe-skip-ca-verification 2022-03-10T03:05:10.359 DEBUG:teuthology.orchestra.run.smithi195:> sudo kubeadm join 172.21.15.37:6443 --node-name smithi195 --token abcdef.20k7qcj4lgeb4yo9 --discovery-token-unsafe-skip-ca-verification 2022-03-10T03:05:10.394 INFO:teuthology.orchestra.run.smithi038.stdout:[preflight] Running pre-flight checks 2022-03-10T03:05:10.401 INFO:teuthology.orchestra.run.smithi195.stdout:[preflight] Running pre-flight checks 2022-03-10T03:05:23.276 INFO:teuthology.orchestra.run.smithi038.stdout:[preflight] Reading configuration from the cluster... 2022-03-10T03:05:23.276 INFO:teuthology.orchestra.run.smithi038.stdout:[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml' 2022-03-10T03:05:23.672 INFO:teuthology.orchestra.run.smithi038.stderr:W0310 03:05:23.681740 10558 utils.go:69] The recommended value for "resolvConf" in "KubeletConfiguration" is: /run/systemd/resolve/resolv.conf; the provided value is: /run/systemd/resolve/resolv.conf 2022-03-10T03:05:23.691 INFO:teuthology.orchestra.run.smithi038.stdout:[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" 2022-03-10T03:05:23.691 INFO:teuthology.orchestra.run.smithi038.stdout:[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" 2022-03-10T03:05:23.691 INFO:teuthology.orchestra.run.smithi038.stdout:[kubelet-start] Starting the kubelet 2022-03-10T03:05:23.917 INFO:teuthology.orchestra.run.smithi038.stdout:[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap... 2022-03-10T03:05:24.008 INFO:teuthology.orchestra.run.smithi195.stdout:[preflight] Reading configuration from the cluster... 2022-03-10T03:05:24.009 INFO:teuthology.orchestra.run.smithi195.stdout:[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml' 2022-03-10T03:05:24.376 INFO:teuthology.orchestra.run.smithi195.stderr:W0310 03:05:24.387133 10693 utils.go:69] The recommended value for "resolvConf" in "KubeletConfiguration" is: /run/systemd/resolve/resolv.conf; the provided value is: /run/systemd/resolve/resolv.conf 2022-03-10T03:05:24.412 INFO:teuthology.orchestra.run.smithi195.stdout:[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" 2022-03-10T03:05:24.412 INFO:teuthology.orchestra.run.smithi195.stdout:[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" 2022-03-10T03:05:24.413 INFO:teuthology.orchestra.run.smithi195.stdout:[kubelet-start] Starting the kubelet 2022-03-10T03:05:24.614 INFO:teuthology.orchestra.run.smithi195.stdout:[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap... 2022-03-10T03:05:29.457 INFO:teuthology.orchestra.run.smithi038.stdout: 2022-03-10T03:05:29.458 INFO:teuthology.orchestra.run.smithi038.stdout:This node has joined the cluster: 2022-03-10T03:05:29.458 INFO:teuthology.orchestra.run.smithi038.stdout:* Certificate signing request was sent to apiserver and a response was received. 2022-03-10T03:05:29.459 INFO:teuthology.orchestra.run.smithi038.stdout:* The Kubelet was informed of the new secure connection details. 2022-03-10T03:05:29.459 INFO:teuthology.orchestra.run.smithi038.stdout: 2022-03-10T03:05:29.460 INFO:teuthology.orchestra.run.smithi038.stdout:Run 'kubectl get nodes' on the control-plane to see this node join the cluster. 2022-03-10T03:05:29.460 INFO:teuthology.orchestra.run.smithi038.stdout: 2022-03-10T03:05:38.132 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-03-10T03:05:38.133 INFO:teuthology.orchestra.run.smithi195.stdout:This node has joined the cluster: 2022-03-10T03:05:38.133 INFO:teuthology.orchestra.run.smithi195.stdout:* Certificate signing request was sent to apiserver and a response was received. 2022-03-10T03:05:38.133 INFO:teuthology.orchestra.run.smithi195.stdout:* The Kubelet was informed of the new secure connection details. 2022-03-10T03:05:38.133 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-03-10T03:05:38.133 INFO:teuthology.orchestra.run.smithi195.stdout:Run 'kubectl get nodes' on the control-plane to see this node join the cluster. 2022-03-10T03:05:38.134 INFO:teuthology.orchestra.run.smithi195.stdout: 2022-03-10T03:05:38.139 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2022-03-10T03:05:38.140 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd if=/etc/kubernetes/admin.conf of=/dev/stdout 2022-03-10T03:05:38.152 INFO:tasks.kubeadm:Setting up kubectl 2022-03-10T03:05:38.153 DEBUG:teuthology.orchestra.run.smithi037:> mkdir -p .kube && sudo mkdir -p /root/.kube 2022-03-10T03:05:38.215 DEBUG:teuthology.orchestra.run.smithi038:> mkdir -p .kube && sudo mkdir -p /root/.kube 2022-03-10T03:05:38.235 DEBUG:teuthology.orchestra.run.smithi195:> mkdir -p .kube && sudo mkdir -p /root/.kube 2022-03-10T03:05:38.253 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2022-03-10T03:05:38.253 DEBUG:teuthology.orchestra.run.smithi037:> dd of=.kube/config 2022-03-10T03:05:38.261 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2022-03-10T03:05:38.261 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd of=/root/.kube/config 2022-03-10T03:05:38.323 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2022-03-10T03:05:38.323 DEBUG:teuthology.orchestra.run.smithi038:> dd of=.kube/config 2022-03-10T03:05:38.330 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2022-03-10T03:05:38.331 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/root/.kube/config 2022-03-10T03:05:38.346 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2022-03-10T03:05:38.346 DEBUG:teuthology.orchestra.run.smithi195:> dd of=.kube/config 2022-03-10T03:05:38.354 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2022-03-10T03:05:38.354 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd of=/root/.kube/config 2022-03-10T03:05:38.365 DEBUG:teuthology.orchestra.run.smithi037:> kubectl apply -f https://docs.projectcalico.org/manifests/tigera-operator.yaml 2022-03-10T03:05:41.111 INFO:teuthology.orchestra.run.smithi037.stdout:customresourcedefinition.apiextensions.k8s.io/bgpconfigurations.crd.projectcalico.org created 2022-03-10T03:05:41.216 INFO:teuthology.orchestra.run.smithi037.stdout:customresourcedefinition.apiextensions.k8s.io/bgppeers.crd.projectcalico.org created 2022-03-10T03:05:41.246 INFO:teuthology.orchestra.run.smithi037.stdout:customresourcedefinition.apiextensions.k8s.io/blockaffinities.crd.projectcalico.org created 2022-03-10T03:05:41.299 INFO:teuthology.orchestra.run.smithi037.stdout:customresourcedefinition.apiextensions.k8s.io/caliconodestatuses.crd.projectcalico.org created 2022-03-10T03:05:41.391 INFO:teuthology.orchestra.run.smithi037.stdout:customresourcedefinition.apiextensions.k8s.io/clusterinformations.crd.projectcalico.org created 2022-03-10T03:05:41.443 INFO:teuthology.orchestra.run.smithi037.stdout:customresourcedefinition.apiextensions.k8s.io/felixconfigurations.crd.projectcalico.org created 2022-03-10T03:05:41.560 INFO:teuthology.orchestra.run.smithi037.stdout:customresourcedefinition.apiextensions.k8s.io/globalnetworkpolicies.crd.projectcalico.org created 2022-03-10T03:05:41.566 INFO:teuthology.orchestra.run.smithi037.stdout:customresourcedefinition.apiextensions.k8s.io/globalnetworksets.crd.projectcalico.org created 2022-03-10T03:05:41.618 INFO:teuthology.orchestra.run.smithi037.stdout:customresourcedefinition.apiextensions.k8s.io/hostendpoints.crd.projectcalico.org created 2022-03-10T03:05:41.722 INFO:teuthology.orchestra.run.smithi037.stdout:customresourcedefinition.apiextensions.k8s.io/ipamblocks.crd.projectcalico.org created 2022-03-10T03:05:41.756 INFO:teuthology.orchestra.run.smithi037.stdout:customresourcedefinition.apiextensions.k8s.io/ipamconfigs.crd.projectcalico.org created 2022-03-10T03:05:41.807 INFO:teuthology.orchestra.run.smithi037.stdout:customresourcedefinition.apiextensions.k8s.io/ipamhandles.crd.projectcalico.org created 2022-03-10T03:05:41.924 INFO:teuthology.orchestra.run.smithi037.stdout:customresourcedefinition.apiextensions.k8s.io/ippools.crd.projectcalico.org created 2022-03-10T03:05:41.974 INFO:teuthology.orchestra.run.smithi037.stdout:customresourcedefinition.apiextensions.k8s.io/ipreservations.crd.projectcalico.org created 2022-03-10T03:05:42.067 INFO:teuthology.orchestra.run.smithi037.stdout:customresourcedefinition.apiextensions.k8s.io/kubecontrollersconfigurations.crd.projectcalico.org created 2022-03-10T03:05:42.120 INFO:teuthology.orchestra.run.smithi037.stdout:customresourcedefinition.apiextensions.k8s.io/networkpolicies.crd.projectcalico.org created 2022-03-10T03:05:42.202 INFO:teuthology.orchestra.run.smithi037.stdout:customresourcedefinition.apiextensions.k8s.io/networksets.crd.projectcalico.org created 2022-03-10T03:05:42.249 INFO:teuthology.orchestra.run.smithi037.stdout:customresourcedefinition.apiextensions.k8s.io/apiservers.operator.tigera.io created 2022-03-10T03:05:42.356 INFO:teuthology.orchestra.run.smithi037.stdout:customresourcedefinition.apiextensions.k8s.io/imagesets.operator.tigera.io created 2022-03-10T03:05:42.422 INFO:teuthology.orchestra.run.smithi037.stdout:customresourcedefinition.apiextensions.k8s.io/installations.operator.tigera.io created 2022-03-10T03:05:42.526 INFO:teuthology.orchestra.run.smithi037.stdout:customresourcedefinition.apiextensions.k8s.io/tigerastatuses.operator.tigera.io created 2022-03-10T03:05:42.583 INFO:teuthology.orchestra.run.smithi037.stdout:namespace/tigera-operator created 2022-03-10T03:05:42.593 INFO:teuthology.orchestra.run.smithi037.stderr:Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+ 2022-03-10T03:05:42.602 INFO:teuthology.orchestra.run.smithi037.stdout:podsecuritypolicy.policy/tigera-operator created 2022-03-10T03:05:42.691 INFO:teuthology.orchestra.run.smithi037.stdout:serviceaccount/tigera-operator created 2022-03-10T03:05:42.706 INFO:teuthology.orchestra.run.smithi037.stdout:clusterrole.rbac.authorization.k8s.io/tigera-operator created 2022-03-10T03:05:42.749 INFO:teuthology.orchestra.run.smithi037.stdout:clusterrolebinding.rbac.authorization.k8s.io/tigera-operator created 2022-03-10T03:05:42.791 INFO:teuthology.orchestra.run.smithi037.stdout:deployment.apps/tigera-operator created 2022-03-10T03:05:42.800 DEBUG:teuthology.orchestra.run.smithi037:> kubectl create -f - 2022-03-10T03:05:45.636 INFO:teuthology.orchestra.run.smithi037.stdout:installation.operator.tigera.io/default created 2022-03-10T03:05:45.644 INFO:tasks.kubeadm:Scanning for scratch devices 2022-03-10T03:05:45.645 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2022-03-10T03:05:45.645 DEBUG:teuthology.orchestra.run.smithi037:> dd if=/scratch_devs of=/dev/stdout 2022-03-10T03:05:45.652 INFO:tasks.kubeadm:Scratch devices on smithi037: ['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-03-10T03:05:45.653 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 2022-03-10T03:05:45.675 INFO:teuthology.orchestra.run.smithi037.stderr:10+0 records in 2022-03-10T03:05:45.675 INFO:teuthology.orchestra.run.smithi037.stderr:10+0 records out 2022-03-10T03:05:45.676 INFO:teuthology.orchestra.run.smithi037.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0130961 s, 801 MB/s 2022-03-10T03:05:45.677 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 2022-03-10T03:05:45.747 INFO:teuthology.orchestra.run.smithi037.stderr:10+0 records in 2022-03-10T03:05:45.747 INFO:teuthology.orchestra.run.smithi037.stderr:10+0 records out 2022-03-10T03:05:45.747 INFO:teuthology.orchestra.run.smithi037.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0138646 s, 756 MB/s 2022-03-10T03:05:45.749 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 2022-03-10T03:05:45.778 INFO:teuthology.orchestra.run.smithi037.stderr:10+0 records in 2022-03-10T03:05:45.778 INFO:teuthology.orchestra.run.smithi037.stderr:10+0 records out 2022-03-10T03:05:45.779 INFO:teuthology.orchestra.run.smithi037.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0166072 s, 631 MB/s 2022-03-10T03:05:45.780 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 2022-03-10T03:05:45.855 INFO:teuthology.orchestra.run.smithi037.stderr:10+0 records in 2022-03-10T03:05:45.856 INFO:teuthology.orchestra.run.smithi037.stderr:10+0 records out 2022-03-10T03:05:45.856 INFO:teuthology.orchestra.run.smithi037.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0179955 s, 583 MB/s 2022-03-10T03:05:45.858 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2022-03-10T03:05:45.859 DEBUG:teuthology.orchestra.run.smithi038:> dd if=/scratch_devs of=/dev/stdout 2022-03-10T03:05:45.867 INFO:tasks.kubeadm:Scratch devices on smithi038: ['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-03-10T03:05:45.868 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 2022-03-10T03:05:45.898 INFO:teuthology.orchestra.run.smithi038.stderr:10+0 records in 2022-03-10T03:05:45.899 INFO:teuthology.orchestra.run.smithi038.stderr:10+0 records out 2022-03-10T03:05:45.899 INFO:teuthology.orchestra.run.smithi038.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0166012 s, 632 MB/s 2022-03-10T03:05:45.900 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 2022-03-10T03:05:45.969 INFO:teuthology.orchestra.run.smithi038.stderr:10+0 records in 2022-03-10T03:05:45.970 INFO:teuthology.orchestra.run.smithi038.stderr:10+0 records out 2022-03-10T03:05:45.970 INFO:teuthology.orchestra.run.smithi038.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0137975 s, 760 MB/s 2022-03-10T03:05:45.971 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 2022-03-10T03:05:45.995 INFO:teuthology.orchestra.run.smithi038.stderr:10+0 records in 2022-03-10T03:05:45.996 INFO:teuthology.orchestra.run.smithi038.stderr:10+0 records out 2022-03-10T03:05:45.996 INFO:teuthology.orchestra.run.smithi038.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0119223 s, 880 MB/s 2022-03-10T03:05:45.997 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 2022-03-10T03:05:46.023 INFO:teuthology.orchestra.run.smithi038.stderr:10+0 records in 2022-03-10T03:05:46.024 INFO:teuthology.orchestra.run.smithi038.stderr:10+0 records out 2022-03-10T03:05:46.024 INFO:teuthology.orchestra.run.smithi038.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0134446 s, 780 MB/s 2022-03-10T03:05:46.025 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2022-03-10T03:05:46.026 DEBUG:teuthology.orchestra.run.smithi195:> dd if=/scratch_devs of=/dev/stdout 2022-03-10T03:05:46.034 INFO:tasks.kubeadm:Scratch devices on smithi195: ['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-03-10T03:05:46.034 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 2022-03-10T03:05:46.102 INFO:teuthology.orchestra.run.smithi195.stderr:10+0 records in 2022-03-10T03:05:46.102 INFO:teuthology.orchestra.run.smithi195.stderr:10+0 records out 2022-03-10T03:05:46.103 INFO:teuthology.orchestra.run.smithi195.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0144039 s, 728 MB/s 2022-03-10T03:05:46.104 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 2022-03-10T03:05:46.128 INFO:teuthology.orchestra.run.smithi195.stderr:10+0 records in 2022-03-10T03:05:46.129 INFO:teuthology.orchestra.run.smithi195.stderr:10+0 records out 2022-03-10T03:05:46.129 INFO:teuthology.orchestra.run.smithi195.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0140437 s, 747 MB/s 2022-03-10T03:05:46.130 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 2022-03-10T03:05:46.201 INFO:teuthology.orchestra.run.smithi195.stderr:10+0 records in 2022-03-10T03:05:46.202 INFO:teuthology.orchestra.run.smithi195.stderr:10+0 records out 2022-03-10T03:05:46.202 INFO:teuthology.orchestra.run.smithi195.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.013026 s, 805 MB/s 2022-03-10T03:05:46.203 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 2022-03-10T03:05:46.230 INFO:teuthology.orchestra.run.smithi195.stderr:10+0 records in 2022-03-10T03:05:46.230 INFO:teuthology.orchestra.run.smithi195.stderr:10+0 records out 2022-03-10T03:05:46.231 INFO:teuthology.orchestra.run.smithi195.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.012812 s, 818 MB/s 2022-03-10T03:05:46.252 INFO:tasks.kubeadm:Creating PVs + StorageClass 2022-03-10T03:05:46.252 DEBUG:tasks.kubeadm:apiVersion: v1 kind: PersistentVolume metadata: name: smithi037-lv-1 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_1 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi037 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi037-lv-2 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_2 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi037 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi037-lv-3 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_3 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi037 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi037-lv-4 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_4 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi037 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi038-lv-1 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_1 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi038 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi038-lv-2 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_2 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi038 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi038-lv-3 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_3 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi038 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi038-lv-4 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_4 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi038 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi195-lv-1 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_1 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi195 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi195-lv-2 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_2 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi195 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi195-lv-3 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_3 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi195 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi195-lv-4 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_4 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi195 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: scratch provisioner: kubernetes.io/no-provisioner volumeBindingMode: WaitForFirstConsumer 2022-03-10T03:05:46.253 DEBUG:teuthology.orchestra.run.smithi037:> kubectl create -f - 2022-03-10T03:05:47.166 INFO:teuthology.orchestra.run.smithi037.stdout:persistentvolume/smithi037-lv-1 created 2022-03-10T03:05:47.196 INFO:teuthology.orchestra.run.smithi037.stdout:persistentvolume/smithi037-lv-2 created 2022-03-10T03:05:47.209 INFO:teuthology.orchestra.run.smithi037.stdout:persistentvolume/smithi037-lv-3 created 2022-03-10T03:05:47.231 INFO:teuthology.orchestra.run.smithi037.stdout:persistentvolume/smithi037-lv-4 created 2022-03-10T03:05:47.271 INFO:teuthology.orchestra.run.smithi037.stdout:persistentvolume/smithi038-lv-1 created 2022-03-10T03:05:47.274 INFO:teuthology.orchestra.run.smithi037.stdout:persistentvolume/smithi038-lv-2 created 2022-03-10T03:05:47.289 INFO:teuthology.orchestra.run.smithi037.stdout:persistentvolume/smithi038-lv-3 created 2022-03-10T03:05:47.301 INFO:teuthology.orchestra.run.smithi037.stdout:persistentvolume/smithi038-lv-4 created 2022-03-10T03:05:47.309 INFO:teuthology.orchestra.run.smithi037.stdout:persistentvolume/smithi195-lv-1 created 2022-03-10T03:05:47.331 INFO:teuthology.orchestra.run.smithi037.stdout:persistentvolume/smithi195-lv-2 created 2022-03-10T03:05:47.343 INFO:teuthology.orchestra.run.smithi037.stdout:persistentvolume/smithi195-lv-3 created 2022-03-10T03:05:47.356 INFO:teuthology.orchestra.run.smithi037.stdout:persistentvolume/smithi195-lv-4 created 2022-03-10T03:05:47.368 INFO:teuthology.orchestra.run.smithi037.stdout:storageclass.storage.k8s.io/scratch created 2022-03-10T03:05:47.374 DEBUG:teuthology.orchestra.run.smithi037:> kubectl taint node smithi037 node-role.kubernetes.io/master- || true 2022-03-10T03:05:47.442 INFO:teuthology.orchestra.run.smithi037.stdout:node/smithi037 untainted 2022-03-10T03:05:47.445 INFO:tasks.kubeadm:Kubeadm complete, yielding 2022-03-10T03:05:47.446 INFO:teuthology.run_tasks:Running task nvme_loop... 2022-03-10T03:05:47.457 INFO:tasks.nvme_loop:Setting up nvme_loop on scratch devices... 2022-03-10T03:05:47.458 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2022-03-10T03:05:47.459 DEBUG:teuthology.orchestra.run.smithi037:> dd if=/scratch_devs of=/dev/stdout 2022-03-10T03:05:47.466 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-03-10T03:05:47.467 DEBUG:teuthology.orchestra.run.smithi037:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2022-03-10T03:05:47.515 INFO:teuthology.orchestra.run.smithi037.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2022-03-10T03:05:47.516 INFO:teuthology.orchestra.run.smithi037.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-10T03:05:47.516 INFO:teuthology.orchestra.run.smithi037.stdout:Device: 5h/5d Inode: 497 Links: 1 2022-03-10T03:05:47.516 INFO:teuthology.orchestra.run.smithi037.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-10T03:05:47.516 INFO:teuthology.orchestra.run.smithi037.stdout:Access: 2022-03-10 03:05:45.687901526 +0000 2022-03-10T03:05:47.517 INFO:teuthology.orchestra.run.smithi037.stdout:Modify: 2022-03-10 03:05:45.683901526 +0000 2022-03-10T03:05:47.517 INFO:teuthology.orchestra.run.smithi037.stdout:Change: 2022-03-10 03:05:45.683901526 +0000 2022-03-10T03:05:47.517 INFO:teuthology.orchestra.run.smithi037.stdout: Birth: - 2022-03-10T03:05:47.523 INFO:teuthology.orchestra.run.smithi037.stderr:1+0 records in 2022-03-10T03:05:47.524 INFO:teuthology.orchestra.run.smithi037.stderr:1+0 records out 2022-03-10T03:05:47.524 INFO:teuthology.orchestra.run.smithi037.stderr:512 bytes copied, 0.000473292 s, 1.1 MB/s 2022-03-10T03:05:47.527 DEBUG:teuthology.orchestra.run.smithi037:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2022-03-10T03:05:47.534 INFO:teuthology.orchestra.run.smithi037.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2022-03-10T03:05:47.535 INFO:teuthology.orchestra.run.smithi037.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-10T03:05:47.535 INFO:teuthology.orchestra.run.smithi037.stdout:Device: 5h/5d Inode: 503 Links: 1 2022-03-10T03:05:47.535 INFO:teuthology.orchestra.run.smithi037.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-10T03:05:47.536 INFO:teuthology.orchestra.run.smithi037.stdout:Access: 2022-03-10 03:05:45.759901524 +0000 2022-03-10T03:05:47.536 INFO:teuthology.orchestra.run.smithi037.stdout:Modify: 2022-03-10 03:05:45.759901524 +0000 2022-03-10T03:05:47.536 INFO:teuthology.orchestra.run.smithi037.stdout:Change: 2022-03-10 03:05:45.759901524 +0000 2022-03-10T03:05:47.536 INFO:teuthology.orchestra.run.smithi037.stdout: Birth: - 2022-03-10T03:05:47.541 INFO:teuthology.orchestra.run.smithi037.stderr:1+0 records in 2022-03-10T03:05:47.541 INFO:teuthology.orchestra.run.smithi037.stderr:1+0 records out 2022-03-10T03:05:47.541 INFO:teuthology.orchestra.run.smithi037.stderr:512 bytes copied, 0.000579226 s, 884 kB/s 2022-03-10T03:05:47.544 DEBUG:teuthology.orchestra.run.smithi037:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2022-03-10T03:05:47.550 INFO:teuthology.orchestra.run.smithi037.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2022-03-10T03:05:47.551 INFO:teuthology.orchestra.run.smithi037.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-10T03:05:47.551 INFO:teuthology.orchestra.run.smithi037.stdout:Device: 5h/5d Inode: 508 Links: 1 2022-03-10T03:05:47.551 INFO:teuthology.orchestra.run.smithi037.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-10T03:05:47.552 INFO:teuthology.orchestra.run.smithi037.stdout:Access: 2022-03-10 03:05:45.791901523 +0000 2022-03-10T03:05:47.552 INFO:teuthology.orchestra.run.smithi037.stdout:Modify: 2022-03-10 03:05:45.787901523 +0000 2022-03-10T03:05:47.552 INFO:teuthology.orchestra.run.smithi037.stdout:Change: 2022-03-10 03:05:45.787901523 +0000 2022-03-10T03:05:47.552 INFO:teuthology.orchestra.run.smithi037.stdout: Birth: - 2022-03-10T03:05:47.559 INFO:teuthology.orchestra.run.smithi037.stderr:1+0 records in 2022-03-10T03:05:47.560 INFO:teuthology.orchestra.run.smithi037.stderr:1+0 records out 2022-03-10T03:05:47.560 INFO:teuthology.orchestra.run.smithi037.stderr:512 bytes copied, 0.000566374 s, 904 kB/s 2022-03-10T03:05:47.565 DEBUG:teuthology.orchestra.run.smithi037:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2022-03-10T03:05:47.615 INFO:teuthology.orchestra.run.smithi037.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2022-03-10T03:05:47.616 INFO:teuthology.orchestra.run.smithi037.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-10T03:05:47.616 INFO:teuthology.orchestra.run.smithi037.stdout:Device: 5h/5d Inode: 514 Links: 1 2022-03-10T03:05:47.616 INFO:teuthology.orchestra.run.smithi037.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-10T03:05:47.617 INFO:teuthology.orchestra.run.smithi037.stdout:Access: 2022-03-10 03:05:45.863901522 +0000 2022-03-10T03:05:47.617 INFO:teuthology.orchestra.run.smithi037.stdout:Modify: 2022-03-10 03:05:45.863901522 +0000 2022-03-10T03:05:47.617 INFO:teuthology.orchestra.run.smithi037.stdout:Change: 2022-03-10 03:05:45.863901522 +0000 2022-03-10T03:05:47.617 INFO:teuthology.orchestra.run.smithi037.stdout: Birth: - 2022-03-10T03:05:47.625 INFO:teuthology.orchestra.run.smithi037.stderr:1+0 records in 2022-03-10T03:05:47.626 INFO:teuthology.orchestra.run.smithi037.stderr:1+0 records out 2022-03-10T03:05:47.626 INFO:teuthology.orchestra.run.smithi037.stderr:512 bytes copied, 0.000495878 s, 1.0 MB/s 2022-03-10T03:05:47.631 DEBUG:teuthology.orchestra.run.smithi037:> sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2022-03-10T03:05:47.715 INFO:teuthology.orchestra.run.smithi037.stdout:loop 2022-03-10T03:05:47.717 INFO:tasks.nvme_loop:Connecting nvme_loop smithi037:/dev/vg_nvme/lv_1... 2022-03-10T03:05:47.717 DEBUG:teuthology.orchestra.run.smithi037:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2022-03-10T03:05:47.743 INFO:teuthology.orchestra.run.smithi037.stdout:1 2022-03-10T03:05:47.769 INFO:teuthology.orchestra.run.smithi037.stdout:/dev/vg_nvme/lv_11 2022-03-10T03:05:47.793 INFO:tasks.nvme_loop:Connecting nvme_loop smithi037:/dev/vg_nvme/lv_2... 2022-03-10T03:05:47.794 DEBUG:teuthology.orchestra.run.smithi037:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2022-03-10T03:05:47.857 INFO:teuthology.orchestra.run.smithi037.stdout:1 2022-03-10T03:05:47.888 INFO:teuthology.orchestra.run.smithi037.stdout:/dev/vg_nvme/lv_21 2022-03-10T03:05:47.911 INFO:tasks.nvme_loop:Connecting nvme_loop smithi037:/dev/vg_nvme/lv_3... 2022-03-10T03:05:47.912 DEBUG:teuthology.orchestra.run.smithi037:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2022-03-10T03:05:47.936 INFO:teuthology.orchestra.run.smithi037.stdout:1 2022-03-10T03:05:47.960 INFO:teuthology.orchestra.run.smithi037.stdout:/dev/vg_nvme/lv_31 2022-03-10T03:05:47.984 INFO:tasks.nvme_loop:Connecting nvme_loop smithi037:/dev/vg_nvme/lv_4... 2022-03-10T03:05:47.984 DEBUG:teuthology.orchestra.run.smithi037:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2022-03-10T03:05:48.003 INFO:teuthology.orchestra.run.smithi037.stdout:1 2022-03-10T03:05:48.029 INFO:teuthology.orchestra.run.smithi037.stdout:/dev/vg_nvme/lv_41 2022-03-10T03:05:48.053 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2022-03-10T03:05:48.053 DEBUG:teuthology.orchestra.run.smithi037:> dd if=/scratch_devs of=/dev/stdout 2022-03-10T03:05:48.103 DEBUG:teuthology.orchestra.run.smithi037:> sudo nvme list 2022-03-10T03:05:48.115 INFO:teuthology.orchestra.run.smithi037.stdout:Node SN Model Namespace Usage Format FW Rev 2022-03-10T03:05:48.116 INFO:teuthology.orchestra.run.smithi037.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2022-03-10T03:05:48.116 INFO:teuthology.orchestra.run.smithi037.stdout:/dev/nvme0n1 CVFT5331005T400BGN INTEL SSDPEDMD400G4 1 400.09 GB / 400.09 GB 512 B + 0 B 8DV101H0 2022-03-10T03:05:48.117 INFO:teuthology.orchestra.run.smithi037.stdout:/dev/nvme1n1 0f8e6670e44d24f9 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-10T03:05:48.117 INFO:teuthology.orchestra.run.smithi037.stdout:/dev/nvme2n1 8e115b7f5cc77a7b Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-10T03:05:48.117 INFO:teuthology.orchestra.run.smithi037.stdout:/dev/nvme3n1 bb61d14db7027518 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-10T03:05:48.117 INFO:teuthology.orchestra.run.smithi037.stdout:/dev/nvme4n1 bfa054601cf669d9 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-10T03:05:48.118 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2022-03-10T03:05:48.119 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2022-03-10T03:05:48.119 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd of=/scratch_devs 2022-03-10T03:05:48.132 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2022-03-10T03:05:48.134 DEBUG:teuthology.orchestra.run.smithi038:> dd if=/scratch_devs of=/dev/stdout 2022-03-10T03:05:48.143 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-03-10T03:05:48.144 DEBUG:teuthology.orchestra.run.smithi038:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2022-03-10T03:05:48.196 INFO:teuthology.orchestra.run.smithi038.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2022-03-10T03:05:48.197 INFO:teuthology.orchestra.run.smithi038.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-10T03:05:48.197 INFO:teuthology.orchestra.run.smithi038.stdout:Device: 5h/5d Inode: 498 Links: 1 2022-03-10T03:05:48.198 INFO:teuthology.orchestra.run.smithi038.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-10T03:05:48.198 INFO:teuthology.orchestra.run.smithi038.stdout:Access: 2022-03-10 03:05:45.907926391 +0000 2022-03-10T03:05:48.198 INFO:teuthology.orchestra.run.smithi038.stdout:Modify: 2022-03-10 03:05:45.907926391 +0000 2022-03-10T03:05:48.199 INFO:teuthology.orchestra.run.smithi038.stdout:Change: 2022-03-10 03:05:45.907926391 +0000 2022-03-10T03:05:48.199 INFO:teuthology.orchestra.run.smithi038.stdout: Birth: - 2022-03-10T03:05:48.206 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records in 2022-03-10T03:05:48.206 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records out 2022-03-10T03:05:48.207 INFO:teuthology.orchestra.run.smithi038.stderr:512 bytes copied, 0.000491064 s, 1.0 MB/s 2022-03-10T03:05:48.211 DEBUG:teuthology.orchestra.run.smithi038:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2022-03-10T03:05:48.219 INFO:teuthology.orchestra.run.smithi038.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2022-03-10T03:05:48.219 INFO:teuthology.orchestra.run.smithi038.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-10T03:05:48.219 INFO:teuthology.orchestra.run.smithi038.stdout:Device: 5h/5d Inode: 507 Links: 1 2022-03-10T03:05:48.219 INFO:teuthology.orchestra.run.smithi038.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-10T03:05:48.220 INFO:teuthology.orchestra.run.smithi038.stdout:Access: 2022-03-10 03:05:45.979926389 +0000 2022-03-10T03:05:48.220 INFO:teuthology.orchestra.run.smithi038.stdout:Modify: 2022-03-10 03:05:45.979926389 +0000 2022-03-10T03:05:48.220 INFO:teuthology.orchestra.run.smithi038.stdout:Change: 2022-03-10 03:05:45.979926389 +0000 2022-03-10T03:05:48.220 INFO:teuthology.orchestra.run.smithi038.stdout: Birth: - 2022-03-10T03:05:48.227 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records in 2022-03-10T03:05:48.227 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records out 2022-03-10T03:05:48.227 INFO:teuthology.orchestra.run.smithi038.stderr:512 bytes copied, 0.000570803 s, 897 kB/s 2022-03-10T03:05:48.231 DEBUG:teuthology.orchestra.run.smithi038:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2022-03-10T03:05:48.237 INFO:teuthology.orchestra.run.smithi038.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2022-03-10T03:05:48.237 INFO:teuthology.orchestra.run.smithi038.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-10T03:05:48.237 INFO:teuthology.orchestra.run.smithi038.stdout:Device: 5h/5d Inode: 513 Links: 1 2022-03-10T03:05:48.238 INFO:teuthology.orchestra.run.smithi038.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-10T03:05:48.238 INFO:teuthology.orchestra.run.smithi038.stdout:Access: 2022-03-10 03:05:46.007926389 +0000 2022-03-10T03:05:48.238 INFO:teuthology.orchestra.run.smithi038.stdout:Modify: 2022-03-10 03:05:46.007926389 +0000 2022-03-10T03:05:48.238 INFO:teuthology.orchestra.run.smithi038.stdout:Change: 2022-03-10 03:05:46.007926389 +0000 2022-03-10T03:05:48.239 INFO:teuthology.orchestra.run.smithi038.stdout: Birth: - 2022-03-10T03:05:48.246 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records in 2022-03-10T03:05:48.246 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records out 2022-03-10T03:05:48.246 INFO:teuthology.orchestra.run.smithi038.stderr:512 bytes copied, 0.000536864 s, 954 kB/s 2022-03-10T03:05:48.250 DEBUG:teuthology.orchestra.run.smithi038:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2022-03-10T03:05:48.258 INFO:teuthology.orchestra.run.smithi038.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2022-03-10T03:05:48.258 INFO:teuthology.orchestra.run.smithi038.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-10T03:05:48.258 INFO:teuthology.orchestra.run.smithi038.stdout:Device: 5h/5d Inode: 517 Links: 1 2022-03-10T03:05:48.258 INFO:teuthology.orchestra.run.smithi038.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-10T03:05:48.259 INFO:teuthology.orchestra.run.smithi038.stdout:Access: 2022-03-10 03:05:46.031926388 +0000 2022-03-10T03:05:48.259 INFO:teuthology.orchestra.run.smithi038.stdout:Modify: 2022-03-10 03:05:46.031926388 +0000 2022-03-10T03:05:48.259 INFO:teuthology.orchestra.run.smithi038.stdout:Change: 2022-03-10 03:05:46.031926388 +0000 2022-03-10T03:05:48.259 INFO:teuthology.orchestra.run.smithi038.stdout: Birth: - 2022-03-10T03:05:48.266 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records in 2022-03-10T03:05:48.266 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records out 2022-03-10T03:05:48.271 INFO:teuthology.orchestra.run.smithi038.stderr:512 bytes copied, 0.00052546 s, 974 kB/s 2022-03-10T03:05:48.272 DEBUG:teuthology.orchestra.run.smithi038:> sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2022-03-10T03:05:48.361 INFO:teuthology.orchestra.run.smithi038.stdout:loop 2022-03-10T03:05:48.363 INFO:tasks.nvme_loop:Connecting nvme_loop smithi038:/dev/vg_nvme/lv_1... 2022-03-10T03:05:48.363 DEBUG:teuthology.orchestra.run.smithi038:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2022-03-10T03:05:48.385 INFO:teuthology.orchestra.run.smithi038.stdout:1 2022-03-10T03:05:48.409 INFO:teuthology.orchestra.run.smithi038.stdout:/dev/vg_nvme/lv_11 2022-03-10T03:05:48.429 INFO:tasks.nvme_loop:Connecting nvme_loop smithi038:/dev/vg_nvme/lv_2... 2022-03-10T03:05:48.430 DEBUG:teuthology.orchestra.run.smithi038:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2022-03-10T03:05:48.454 INFO:teuthology.orchestra.run.smithi038.stdout:1 2022-03-10T03:05:48.477 INFO:teuthology.orchestra.run.smithi038.stdout:/dev/vg_nvme/lv_21 2022-03-10T03:05:48.501 INFO:tasks.nvme_loop:Connecting nvme_loop smithi038:/dev/vg_nvme/lv_3... 2022-03-10T03:05:48.501 DEBUG:teuthology.orchestra.run.smithi038:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2022-03-10T03:05:48.523 INFO:teuthology.orchestra.run.smithi038.stdout:1 2022-03-10T03:05:48.546 INFO:teuthology.orchestra.run.smithi038.stdout:/dev/vg_nvme/lv_31 2022-03-10T03:05:48.566 INFO:tasks.nvme_loop:Connecting nvme_loop smithi038:/dev/vg_nvme/lv_4... 2022-03-10T03:05:48.567 DEBUG:teuthology.orchestra.run.smithi038:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2022-03-10T03:05:48.631 INFO:teuthology.orchestra.run.smithi038.stdout:1 2022-03-10T03:05:48.651 INFO:teuthology.orchestra.run.smithi038.stdout:/dev/vg_nvme/lv_41 2022-03-10T03:05:48.669 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2022-03-10T03:05:48.669 DEBUG:teuthology.orchestra.run.smithi038:> dd if=/scratch_devs of=/dev/stdout 2022-03-10T03:05:48.675 DEBUG:teuthology.orchestra.run.smithi038:> sudo nvme list 2022-03-10T03:05:48.687 INFO:teuthology.orchestra.run.smithi038.stdout:Node SN Model Namespace Usage Format FW Rev 2022-03-10T03:05:48.688 INFO:teuthology.orchestra.run.smithi038.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2022-03-10T03:05:48.688 INFO:teuthology.orchestra.run.smithi038.stdout:/dev/nvme0n1 BTLE736103BB4P0KGN INTEL SSDPEDKE040T7 1 4.00 TB / 4.00 TB 512 B + 0 B QDV1013D 2022-03-10T03:05:48.688 INFO:teuthology.orchestra.run.smithi038.stdout:/dev/nvme1n1 11861a117cc6b814 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-10T03:05:48.688 INFO:teuthology.orchestra.run.smithi038.stdout:/dev/nvme2n1 990b19f4c1813965 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-10T03:05:48.689 INFO:teuthology.orchestra.run.smithi038.stdout:/dev/nvme3n1 39f5a35b239d71c7 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-10T03:05:48.690 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1'] 2022-03-10T03:05:49.691 DEBUG:teuthology.orchestra.run.smithi038:> sudo nvme list 2022-03-10T03:05:49.705 INFO:teuthology.orchestra.run.smithi038.stdout:Node SN Model Namespace Usage Format FW Rev 2022-03-10T03:05:49.705 INFO:teuthology.orchestra.run.smithi038.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2022-03-10T03:05:49.705 INFO:teuthology.orchestra.run.smithi038.stdout:/dev/nvme0n1 BTLE736103BB4P0KGN INTEL SSDPEDKE040T7 1 4.00 TB / 4.00 TB 512 B + 0 B QDV1013D 2022-03-10T03:05:49.706 INFO:teuthology.orchestra.run.smithi038.stdout:/dev/nvme1n1 11861a117cc6b814 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-10T03:05:49.706 INFO:teuthology.orchestra.run.smithi038.stdout:/dev/nvme2n1 990b19f4c1813965 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-10T03:05:49.706 INFO:teuthology.orchestra.run.smithi038.stdout:/dev/nvme3n1 39f5a35b239d71c7 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-10T03:05:49.706 INFO:teuthology.orchestra.run.smithi038.stdout:/dev/nvme4n1 4b1c347621445701 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-10T03:05:49.707 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2022-03-10T03:05:49.708 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2022-03-10T03:05:49.708 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/scratch_devs 2022-03-10T03:05:49.763 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2022-03-10T03:05:49.763 DEBUG:teuthology.orchestra.run.smithi195:> dd if=/scratch_devs of=/dev/stdout 2022-03-10T03:05:49.771 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-03-10T03:05:49.772 DEBUG:teuthology.orchestra.run.smithi195:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2022-03-10T03:05:49.779 INFO:teuthology.orchestra.run.smithi195.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2022-03-10T03:05:49.779 INFO:teuthology.orchestra.run.smithi195.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-10T03:05:49.779 INFO:teuthology.orchestra.run.smithi195.stdout:Device: 5h/5d Inode: 498 Links: 1 2022-03-10T03:05:49.779 INFO:teuthology.orchestra.run.smithi195.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-10T03:05:49.780 INFO:teuthology.orchestra.run.smithi195.stdout:Access: 2022-03-10 03:05:46.111437966 +0000 2022-03-10T03:05:49.780 INFO:teuthology.orchestra.run.smithi195.stdout:Modify: 2022-03-10 03:05:46.111437966 +0000 2022-03-10T03:05:49.780 INFO:teuthology.orchestra.run.smithi195.stdout:Change: 2022-03-10 03:05:46.111437966 +0000 2022-03-10T03:05:49.780 INFO:teuthology.orchestra.run.smithi195.stdout: Birth: - 2022-03-10T03:05:49.787 INFO:teuthology.orchestra.run.smithi195.stderr:1+0 records in 2022-03-10T03:05:49.788 INFO:teuthology.orchestra.run.smithi195.stderr:1+0 records out 2022-03-10T03:05:49.788 INFO:teuthology.orchestra.run.smithi195.stderr:512 bytes copied, 0.000226273 s, 2.3 MB/s 2022-03-10T03:05:49.793 DEBUG:teuthology.orchestra.run.smithi195:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2022-03-10T03:05:49.842 INFO:teuthology.orchestra.run.smithi195.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2022-03-10T03:05:49.842 INFO:teuthology.orchestra.run.smithi195.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-10T03:05:49.843 INFO:teuthology.orchestra.run.smithi195.stdout:Device: 5h/5d Inode: 507 Links: 1 2022-03-10T03:05:49.843 INFO:teuthology.orchestra.run.smithi195.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-10T03:05:49.843 INFO:teuthology.orchestra.run.smithi195.stdout:Access: 2022-03-10 03:05:46.139437967 +0000 2022-03-10T03:05:49.843 INFO:teuthology.orchestra.run.smithi195.stdout:Modify: 2022-03-10 03:05:46.139437967 +0000 2022-03-10T03:05:49.844 INFO:teuthology.orchestra.run.smithi195.stdout:Change: 2022-03-10 03:05:46.139437967 +0000 2022-03-10T03:05:49.844 INFO:teuthology.orchestra.run.smithi195.stdout: Birth: - 2022-03-10T03:05:49.851 INFO:teuthology.orchestra.run.smithi195.stderr:1+0 records in 2022-03-10T03:05:49.851 INFO:teuthology.orchestra.run.smithi195.stderr:1+0 records out 2022-03-10T03:05:49.851 INFO:teuthology.orchestra.run.smithi195.stderr:512 bytes copied, 0.000279887 s, 1.8 MB/s 2022-03-10T03:05:49.856 DEBUG:teuthology.orchestra.run.smithi195:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2022-03-10T03:05:49.861 INFO:teuthology.orchestra.run.smithi195.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2022-03-10T03:05:49.861 INFO:teuthology.orchestra.run.smithi195.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-10T03:05:49.862 INFO:teuthology.orchestra.run.smithi195.stdout:Device: 5h/5d Inode: 511 Links: 1 2022-03-10T03:05:49.862 INFO:teuthology.orchestra.run.smithi195.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-10T03:05:49.862 INFO:teuthology.orchestra.run.smithi195.stdout:Access: 2022-03-10 03:05:46.215437969 +0000 2022-03-10T03:05:49.863 INFO:teuthology.orchestra.run.smithi195.stdout:Modify: 2022-03-10 03:05:46.211437969 +0000 2022-03-10T03:05:49.863 INFO:teuthology.orchestra.run.smithi195.stdout:Change: 2022-03-10 03:05:46.211437969 +0000 2022-03-10T03:05:49.863 INFO:teuthology.orchestra.run.smithi195.stdout: Birth: - 2022-03-10T03:05:49.869 INFO:teuthology.orchestra.run.smithi195.stderr:1+0 records in 2022-03-10T03:05:49.869 INFO:teuthology.orchestra.run.smithi195.stderr:1+0 records out 2022-03-10T03:05:49.869 INFO:teuthology.orchestra.run.smithi195.stderr:512 bytes copied, 0.000262541 s, 2.0 MB/s 2022-03-10T03:05:49.873 DEBUG:teuthology.orchestra.run.smithi195:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2022-03-10T03:05:49.881 INFO:teuthology.orchestra.run.smithi195.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2022-03-10T03:05:49.881 INFO:teuthology.orchestra.run.smithi195.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-10T03:05:49.881 INFO:teuthology.orchestra.run.smithi195.stdout:Device: 5h/5d Inode: 517 Links: 1 2022-03-10T03:05:49.881 INFO:teuthology.orchestra.run.smithi195.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-10T03:05:49.882 INFO:teuthology.orchestra.run.smithi195.stdout:Access: 2022-03-10 03:05:46.243437969 +0000 2022-03-10T03:05:49.882 INFO:teuthology.orchestra.run.smithi195.stdout:Modify: 2022-03-10 03:05:46.239437969 +0000 2022-03-10T03:05:49.882 INFO:teuthology.orchestra.run.smithi195.stdout:Change: 2022-03-10 03:05:46.239437969 +0000 2022-03-10T03:05:49.882 INFO:teuthology.orchestra.run.smithi195.stdout: Birth: - 2022-03-10T03:05:49.890 INFO:teuthology.orchestra.run.smithi195.stderr:1+0 records in 2022-03-10T03:05:49.890 INFO:teuthology.orchestra.run.smithi195.stderr:1+0 records out 2022-03-10T03:05:49.890 INFO:teuthology.orchestra.run.smithi195.stderr:512 bytes copied, 0.000260503 s, 2.0 MB/s 2022-03-10T03:05:49.894 DEBUG:teuthology.orchestra.run.smithi195:> sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2022-03-10T03:05:50.222 INFO:teuthology.orchestra.run.smithi195.stdout:loop 2022-03-10T03:05:50.223 INFO:tasks.nvme_loop:Connecting nvme_loop smithi195:/dev/vg_nvme/lv_1... 2022-03-10T03:05:50.224 DEBUG:teuthology.orchestra.run.smithi195:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2022-03-10T03:05:50.248 INFO:teuthology.orchestra.run.smithi195.stdout:1 2022-03-10T03:05:50.273 INFO:teuthology.orchestra.run.smithi195.stdout:/dev/vg_nvme/lv_11 2022-03-10T03:05:50.299 INFO:tasks.nvme_loop:Connecting nvme_loop smithi195:/dev/vg_nvme/lv_2... 2022-03-10T03:05:50.300 DEBUG:teuthology.orchestra.run.smithi195:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2022-03-10T03:05:50.321 INFO:teuthology.orchestra.run.smithi195.stdout:1 2022-03-10T03:05:50.347 INFO:teuthology.orchestra.run.smithi195.stdout:/dev/vg_nvme/lv_21 2022-03-10T03:05:50.370 INFO:tasks.nvme_loop:Connecting nvme_loop smithi195:/dev/vg_nvme/lv_3... 2022-03-10T03:05:50.371 DEBUG:teuthology.orchestra.run.smithi195:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2022-03-10T03:05:50.436 INFO:teuthology.orchestra.run.smithi195.stdout:1 2022-03-10T03:05:50.461 INFO:teuthology.orchestra.run.smithi195.stdout:/dev/vg_nvme/lv_31 2022-03-10T03:05:50.481 INFO:tasks.nvme_loop:Connecting nvme_loop smithi195:/dev/vg_nvme/lv_4... 2022-03-10T03:05:50.482 DEBUG:teuthology.orchestra.run.smithi195:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2022-03-10T03:05:50.503 INFO:teuthology.orchestra.run.smithi195.stdout:1 2022-03-10T03:05:50.529 INFO:teuthology.orchestra.run.smithi195.stdout:/dev/vg_nvme/lv_41 2022-03-10T03:05:50.553 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2022-03-10T03:05:50.553 DEBUG:teuthology.orchestra.run.smithi195:> dd if=/scratch_devs of=/dev/stdout 2022-03-10T03:05:50.560 DEBUG:teuthology.orchestra.run.smithi195:> sudo nvme list 2022-03-10T03:05:50.571 INFO:teuthology.orchestra.run.smithi195.stdout:Node SN Model Namespace Usage Format FW Rev 2022-03-10T03:05:50.572 INFO:teuthology.orchestra.run.smithi195.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2022-03-10T03:05:50.572 INFO:teuthology.orchestra.run.smithi195.stdout:/dev/nvme0n1 PHMB751300FS480DGN INTEL SSDPED1D480GA 1 480.10 GB / 480.10 GB 512 B + 0 B E2010325 2022-03-10T03:05:50.572 INFO:teuthology.orchestra.run.smithi195.stdout:/dev/nvme1n1 aa9336a7ba602afa Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-10T03:05:50.572 INFO:teuthology.orchestra.run.smithi195.stdout:/dev/nvme2n1 07bb9f8af8ba3d2a Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-10T03:05:50.573 INFO:teuthology.orchestra.run.smithi195.stdout:/dev/nvme3n1 197ffc9805617b01 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-10T03:05:50.574 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1'] 2022-03-10T03:05:51.575 DEBUG:teuthology.orchestra.run.smithi195:> sudo nvme list 2022-03-10T03:05:51.591 INFO:teuthology.orchestra.run.smithi195.stdout:Node SN Model Namespace Usage Format FW Rev 2022-03-10T03:05:51.591 INFO:teuthology.orchestra.run.smithi195.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2022-03-10T03:05:51.591 INFO:teuthology.orchestra.run.smithi195.stdout:/dev/nvme0n1 PHMB751300FS480DGN INTEL SSDPED1D480GA 1 480.10 GB / 480.10 GB 512 B + 0 B E2010325 2022-03-10T03:05:51.592 INFO:teuthology.orchestra.run.smithi195.stdout:/dev/nvme1n1 aa9336a7ba602afa Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-10T03:05:51.592 INFO:teuthology.orchestra.run.smithi195.stdout:/dev/nvme2n1 07bb9f8af8ba3d2a Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-10T03:05:51.592 INFO:teuthology.orchestra.run.smithi195.stdout:/dev/nvme3n1 197ffc9805617b01 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-10T03:05:51.592 INFO:teuthology.orchestra.run.smithi195.stdout:/dev/nvme4n1 aad86fd2a5340b57 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-10T03:05:51.594 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2022-03-10T03:05:51.594 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2022-03-10T03:05:51.594 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd of=/scratch_devs 2022-03-10T03:05:51.651 INFO:teuthology.run_tasks:Running task rook... 2022-03-10T03:05:51.724 INFO:tasks.rook:Rook start 2022-03-10T03:05:51.725 INFO:tasks.rook:Config: {'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '7a44e9ebca556862e9704be295a07719d7db47b3', 'rook_image': 'rook/ceph:master'} 2022-03-10T03:05:51.725 INFO:tasks.rook:Ceph image is quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3 2022-03-10T03:05:51.725 INFO:tasks.rook:Cloning https://github.com/rook/rook branch master 2022-03-10T03:05:51.726 DEBUG:teuthology.orchestra.run.smithi037:> rm -rf rook && git clone --single-branch --branch master https://github.com/rook/rook rook 2022-03-10T03:05:51.736 INFO:teuthology.orchestra.run.smithi037.stderr:Cloning into 'rook'... 2022-03-10T03:05:54.475 INFO:tasks.rook:/home/teuthworker 2022-03-10T03:05:54.476 DEBUG:teuthology.orchestra.run.smithi037:> PATH=/sbin:/usr/sbin ip addr show 2022-03-10T03:05:54.482 INFO:teuthology.orchestra.run.smithi037.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2022-03-10T03:05:54.483 INFO:teuthology.orchestra.run.smithi037.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2022-03-10T03:05:54.483 INFO:teuthology.orchestra.run.smithi037.stdout: inet 127.0.0.1/8 scope host lo 2022-03-10T03:05:54.483 INFO:teuthology.orchestra.run.smithi037.stdout: valid_lft forever preferred_lft forever 2022-03-10T03:05:54.483 INFO:teuthology.orchestra.run.smithi037.stdout: inet6 ::1/128 scope host 2022-03-10T03:05:54.484 INFO:teuthology.orchestra.run.smithi037.stdout: valid_lft forever preferred_lft forever 2022-03-10T03:05:54.484 INFO:teuthology.orchestra.run.smithi037.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2022-03-10T03:05:54.484 INFO:teuthology.orchestra.run.smithi037.stdout: link/ether 0c:c4:7a:6b:fd:18 brd ff:ff:ff:ff:ff:ff 2022-03-10T03:05:54.484 INFO:teuthology.orchestra.run.smithi037.stdout: altname enp7s0f0 2022-03-10T03:05:54.485 INFO:teuthology.orchestra.run.smithi037.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2022-03-10T03:05:54.485 INFO:teuthology.orchestra.run.smithi037.stdout: link/ether 0c:c4:7a:6b:fd:19 brd ff:ff:ff:ff:ff:ff 2022-03-10T03:05:54.485 INFO:teuthology.orchestra.run.smithi037.stdout: altname enp7s0f1 2022-03-10T03:05:54.485 INFO:teuthology.orchestra.run.smithi037.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2022-03-10T03:05:54.486 INFO:teuthology.orchestra.run.smithi037.stdout: link/ether 0c:c4:7a:88:77:94 brd ff:ff:ff:ff:ff:ff 2022-03-10T03:05:54.486 INFO:teuthology.orchestra.run.smithi037.stdout: altname enp1s0f0 2022-03-10T03:05:54.486 INFO:teuthology.orchestra.run.smithi037.stdout: inet 172.21.15.37/20 brd 172.21.15.255 scope global dynamic ens1f0 2022-03-10T03:05:54.486 INFO:teuthology.orchestra.run.smithi037.stdout: valid_lft 42867sec preferred_lft 42867sec 2022-03-10T03:05:54.487 INFO:teuthology.orchestra.run.smithi037.stdout: inet6 fe80::ec4:7aff:fe88:7794/64 scope link 2022-03-10T03:05:54.487 INFO:teuthology.orchestra.run.smithi037.stdout: valid_lft forever preferred_lft forever 2022-03-10T03:05:54.487 INFO:teuthology.orchestra.run.smithi037.stdout:5: ens1f1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2022-03-10T03:05:54.487 INFO:teuthology.orchestra.run.smithi037.stdout: link/ether 0c:c4:7a:88:77:95 brd ff:ff:ff:ff:ff:ff 2022-03-10T03:05:54.488 INFO:teuthology.orchestra.run.smithi037.stdout: altname enp1s0f1 2022-03-10T03:05:54.488 INFO:teuthology.orchestra.run.smithi037.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2022-03-10T03:05:54.488 INFO:teuthology.orchestra.run.smithi037.stdout: link/ether 02:42:a8:f7:2e:86 brd ff:ff:ff:ff:ff:ff 2022-03-10T03:05:54.488 INFO:teuthology.orchestra.run.smithi037.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2022-03-10T03:05:54.488 INFO:teuthology.orchestra.run.smithi037.stdout: valid_lft forever preferred_lft forever 2022-03-10T03:05:54.490 DEBUG:teuthology.orchestra.run.smithi037:> which systemctl 2022-03-10T03:05:54.496 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/systemctl 2022-03-10T03:05:54.496 DEBUG:teuthology.orchestra.run.smithi037:> test -f /run/.containerenv -o -f /.dockerenv 2022-03-10T03:05:54.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-03-10T03:05:54.542 DEBUG:teuthology.orchestra.run.smithi037:> true 2022-03-10T03:05:54.577 DEBUG:teuthology.orchestra.run.smithi037:> sudo lsb_release -is 2022-03-10T03:05:54.625 INFO:teuthology.orchestra.run.smithi037.stdout:Ubuntu 2022-03-10T03:05:54.631 DEBUG:teuthology.misc:System to be installed: Ubuntu 2022-03-10T03:05:54.632 INFO:tasks.rook:['__class__', '__del__', '__delattr__', '__dir__', '__eq__', '__format__', '__ge__', '__getattribute__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__le__', '__lt__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '_format_size', '_reconnect', '_runner', '_set_iface_and_cidr', '_sftp_get_file', '_sftp_get_size', '_sftp_open_file', '_sftp_put_file', 'chcon', 'chmod', 'connect', 'copy_file', 'ensure_online', 'get_file', 'get_tar', 'get_tar_stream', 'mkdtemp', 'mktemp', 'move_file', 'put_file', 'read_file', 'reconnect', 'remove', 'run', 'sh', 'sh_file', 'sudo_write_file', 'write_file'] 2022-03-10T03:05:54.641 DEBUG:teuthology.orchestra.remote:smithi037:rook/deploy/examples/operator.yaml is 20KB 2022-03-10T03:05:54.698 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2022-03-10T03:05:54.699 DEBUG:teuthology.orchestra.run.smithi037:> dd if=rook/deploy/examples/operator.yaml of=/dev/stdout 2022-03-10T03:05:54.707 INFO:tasks.rook:Patching operator to use image rook/ceph:master 2022-03-10T03:05:54.748 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2022-03-10T03:05:54.748 DEBUG:teuthology.orchestra.run.smithi037:> dd of=operator.yaml 2022-03-10T03:05:54.756 INFO:tasks.rook:Deploying operator 2022-03-10T03:05:54.766 DEBUG:teuthology.orchestra.remote:smithi037:rook/deploy/examples/operator.yaml is 20KB 2022-03-10T03:05:54.826 DEBUG:teuthology.orchestra.remote:smithi037:rook/deploy/examples/operator.yaml is 20KB 2022-03-10T03:05:54.840 DEBUG:teuthology.orchestra.run.smithi037:> kubectl create -f rook/deploy/examples/crds.yaml -f rook/deploy/examples/common.yaml -f operator.yaml 2022-03-10T03:05:55.111 INFO:teuthology.orchestra.run.smithi037.stdout:customresourcedefinition.apiextensions.k8s.io/cephblockpools.ceph.rook.io created 2022-03-10T03:05:55.122 INFO:teuthology.orchestra.run.smithi037.stdout:customresourcedefinition.apiextensions.k8s.io/cephbucketnotifications.ceph.rook.io created 2022-03-10T03:05:55.148 INFO:teuthology.orchestra.run.smithi037.stdout:customresourcedefinition.apiextensions.k8s.io/cephbuckettopics.ceph.rook.io created 2022-03-10T03:05:55.173 INFO:teuthology.orchestra.run.smithi037.stdout:customresourcedefinition.apiextensions.k8s.io/cephclients.ceph.rook.io created 2022-03-10T03:05:55.289 INFO:teuthology.orchestra.run.smithi037.stdout:customresourcedefinition.apiextensions.k8s.io/cephclusters.ceph.rook.io created 2022-03-10T03:05:55.314 INFO:teuthology.orchestra.run.smithi037.stdout:customresourcedefinition.apiextensions.k8s.io/cephfilesystemmirrors.ceph.rook.io created 2022-03-10T03:05:55.432 INFO:teuthology.orchestra.run.smithi037.stdout:customresourcedefinition.apiextensions.k8s.io/cephfilesystems.ceph.rook.io created 2022-03-10T03:05:55.451 INFO:teuthology.orchestra.run.smithi037.stdout:customresourcedefinition.apiextensions.k8s.io/cephfilesystemsubvolumegroups.ceph.rook.io created 2022-03-10T03:05:55.477 INFO:teuthology.orchestra.run.smithi037.stdout:customresourcedefinition.apiextensions.k8s.io/cephnfses.ceph.rook.io created 2022-03-10T03:05:55.498 INFO:teuthology.orchestra.run.smithi037.stdout:customresourcedefinition.apiextensions.k8s.io/cephobjectrealms.ceph.rook.io created 2022-03-10T03:05:55.635 INFO:teuthology.orchestra.run.smithi037.stdout:customresourcedefinition.apiextensions.k8s.io/cephobjectstores.ceph.rook.io created 2022-03-10T03:05:55.708 INFO:teuthology.orchestra.run.smithi037.stdout:customresourcedefinition.apiextensions.k8s.io/cephobjectstoreusers.ceph.rook.io created 2022-03-10T03:05:55.832 INFO:teuthology.orchestra.run.smithi037.stdout:customresourcedefinition.apiextensions.k8s.io/cephobjectzonegroups.ceph.rook.io created 2022-03-10T03:05:55.917 INFO:teuthology.orchestra.run.smithi037.stdout:customresourcedefinition.apiextensions.k8s.io/cephobjectzones.ceph.rook.io created 2022-03-10T03:05:55.990 INFO:teuthology.orchestra.run.smithi037.stdout:customresourcedefinition.apiextensions.k8s.io/cephrbdmirrors.ceph.rook.io created 2022-03-10T03:05:56.007 INFO:teuthology.orchestra.run.smithi037.stdout:customresourcedefinition.apiextensions.k8s.io/objectbucketclaims.objectbucket.io created 2022-03-10T03:05:56.033 INFO:teuthology.orchestra.run.smithi037.stdout:customresourcedefinition.apiextensions.k8s.io/objectbuckets.objectbucket.io created 2022-03-10T03:05:56.139 INFO:teuthology.orchestra.run.smithi037.stdout:namespace/rook-ceph created 2022-03-10T03:05:56.166 INFO:teuthology.orchestra.run.smithi037.stdout:clusterrole.rbac.authorization.k8s.io/cephfs-csi-nodeplugin created 2022-03-10T03:05:56.191 INFO:teuthology.orchestra.run.smithi037.stdout:clusterrole.rbac.authorization.k8s.io/cephfs-external-provisioner-runner created 2022-03-10T03:05:56.216 INFO:teuthology.orchestra.run.smithi037.stdout:clusterrole.rbac.authorization.k8s.io/psp:rook created 2022-03-10T03:05:56.304 INFO:teuthology.orchestra.run.smithi037.stdout:clusterrole.rbac.authorization.k8s.io/rbd-csi-nodeplugin created 2022-03-10T03:05:56.316 INFO:teuthology.orchestra.run.smithi037.stdout:clusterrole.rbac.authorization.k8s.io/rbd-external-provisioner-runner created 2022-03-10T03:05:56.342 INFO:teuthology.orchestra.run.smithi037.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-cluster-mgmt created 2022-03-10T03:05:56.349 INFO:teuthology.orchestra.run.smithi037.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-global created 2022-03-10T03:05:56.375 INFO:teuthology.orchestra.run.smithi037.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-mgr-cluster created 2022-03-10T03:05:56.382 INFO:teuthology.orchestra.run.smithi037.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-mgr-system created 2022-03-10T03:05:56.391 INFO:teuthology.orchestra.run.smithi037.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-object-bucket created 2022-03-10T03:05:56.445 INFO:teuthology.orchestra.run.smithi037.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-osd created 2022-03-10T03:05:56.449 INFO:teuthology.orchestra.run.smithi037.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-system created 2022-03-10T03:05:56.457 INFO:teuthology.orchestra.run.smithi037.stdout:clusterrolebinding.rbac.authorization.k8s.io/cephfs-csi-nodeplugin created 2022-03-10T03:05:56.466 INFO:teuthology.orchestra.run.smithi037.stdout:clusterrolebinding.rbac.authorization.k8s.io/cephfs-csi-provisioner-role created 2022-03-10T03:05:56.474 INFO:teuthology.orchestra.run.smithi037.stdout:clusterrolebinding.rbac.authorization.k8s.io/rbd-csi-nodeplugin created 2022-03-10T03:05:56.483 INFO:teuthology.orchestra.run.smithi037.stdout:clusterrolebinding.rbac.authorization.k8s.io/rbd-csi-provisioner-role created 2022-03-10T03:05:56.491 INFO:teuthology.orchestra.run.smithi037.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-global created 2022-03-10T03:05:56.517 INFO:teuthology.orchestra.run.smithi037.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-mgr-cluster created 2022-03-10T03:05:56.541 INFO:teuthology.orchestra.run.smithi037.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-object-bucket created 2022-03-10T03:05:56.586 INFO:teuthology.orchestra.run.smithi037.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-osd created 2022-03-10T03:05:56.591 INFO:teuthology.orchestra.run.smithi037.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-system created 2022-03-10T03:05:56.599 INFO:teuthology.orchestra.run.smithi037.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-system-psp created 2022-03-10T03:05:56.607 INFO:teuthology.orchestra.run.smithi037.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-csi-cephfs-plugin-sa-psp created 2022-03-10T03:05:56.633 INFO:teuthology.orchestra.run.smithi037.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-csi-cephfs-provisioner-sa-psp created 2022-03-10T03:05:56.641 INFO:teuthology.orchestra.run.smithi037.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-csi-rbd-plugin-sa-psp created 2022-03-10T03:05:56.649 INFO:teuthology.orchestra.run.smithi037.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-csi-rbd-provisioner-sa-psp created 2022-03-10T03:05:56.657 INFO:teuthology.orchestra.run.smithi037.stderr:Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+ 2022-03-10T03:05:56.658 INFO:teuthology.orchestra.run.smithi037.stdout:podsecuritypolicy.policy/00-rook-privileged created 2022-03-10T03:05:56.666 INFO:teuthology.orchestra.run.smithi037.stdout:role.rbac.authorization.k8s.io/cephfs-external-provisioner-cfg created 2022-03-10T03:05:56.712 INFO:teuthology.orchestra.run.smithi037.stdout:role.rbac.authorization.k8s.io/rbd-csi-nodeplugin created 2022-03-10T03:05:56.716 INFO:teuthology.orchestra.run.smithi037.stdout:role.rbac.authorization.k8s.io/rbd-external-provisioner-cfg created 2022-03-10T03:05:56.724 INFO:teuthology.orchestra.run.smithi037.stdout:role.rbac.authorization.k8s.io/rook-ceph-cmd-reporter created 2022-03-10T03:05:56.733 INFO:teuthology.orchestra.run.smithi037.stdout:role.rbac.authorization.k8s.io/rook-ceph-mgr created 2022-03-10T03:05:56.741 INFO:teuthology.orchestra.run.smithi037.stdout:role.rbac.authorization.k8s.io/rook-ceph-osd created 2022-03-10T03:05:56.749 INFO:teuthology.orchestra.run.smithi037.stdout:role.rbac.authorization.k8s.io/rook-ceph-purge-osd created 2022-03-10T03:05:56.758 INFO:teuthology.orchestra.run.smithi037.stdout:role.rbac.authorization.k8s.io/rook-ceph-system created 2022-03-10T03:05:56.783 INFO:teuthology.orchestra.run.smithi037.stdout:rolebinding.rbac.authorization.k8s.io/cephfs-csi-provisioner-role-cfg created 2022-03-10T03:05:56.791 INFO:teuthology.orchestra.run.smithi037.stdout:rolebinding.rbac.authorization.k8s.io/rbd-csi-nodeplugin-role-cfg created 2022-03-10T03:05:56.799 INFO:teuthology.orchestra.run.smithi037.stdout:rolebinding.rbac.authorization.k8s.io/rbd-csi-provisioner-role-cfg created 2022-03-10T03:05:56.807 INFO:teuthology.orchestra.run.smithi037.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-cluster-mgmt created 2022-03-10T03:05:56.845 INFO:teuthology.orchestra.run.smithi037.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-cmd-reporter created 2022-03-10T03:05:56.848 INFO:teuthology.orchestra.run.smithi037.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-cmd-reporter-psp created 2022-03-10T03:05:56.875 INFO:teuthology.orchestra.run.smithi037.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-default-psp created 2022-03-10T03:05:56.882 INFO:teuthology.orchestra.run.smithi037.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-mgr created 2022-03-10T03:05:56.891 INFO:teuthology.orchestra.run.smithi037.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-mgr-psp created 2022-03-10T03:05:56.899 INFO:teuthology.orchestra.run.smithi037.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-mgr-system created 2022-03-10T03:05:56.907 INFO:teuthology.orchestra.run.smithi037.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-osd created 2022-03-10T03:05:56.916 INFO:teuthology.orchestra.run.smithi037.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-osd-psp created 2022-03-10T03:05:56.943 INFO:teuthology.orchestra.run.smithi037.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-purge-osd created 2022-03-10T03:05:56.992 INFO:teuthology.orchestra.run.smithi037.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-system created 2022-03-10T03:05:57.000 INFO:teuthology.orchestra.run.smithi037.stdout:serviceaccount/rook-ceph-cmd-reporter created 2022-03-10T03:05:57.007 INFO:teuthology.orchestra.run.smithi037.stdout:serviceaccount/rook-ceph-mgr created 2022-03-10T03:05:57.016 INFO:teuthology.orchestra.run.smithi037.stdout:serviceaccount/rook-ceph-osd created 2022-03-10T03:05:57.042 INFO:teuthology.orchestra.run.smithi037.stdout:serviceaccount/rook-ceph-purge-osd created 2022-03-10T03:05:57.068 INFO:teuthology.orchestra.run.smithi037.stdout:serviceaccount/rook-ceph-system created 2022-03-10T03:05:57.137 INFO:teuthology.orchestra.run.smithi037.stdout:serviceaccount/rook-csi-cephfs-plugin-sa created 2022-03-10T03:05:57.150 INFO:teuthology.orchestra.run.smithi037.stdout:serviceaccount/rook-csi-cephfs-provisioner-sa created 2022-03-10T03:05:57.177 INFO:teuthology.orchestra.run.smithi037.stdout:serviceaccount/rook-csi-rbd-plugin-sa created 2022-03-10T03:05:57.201 INFO:teuthology.orchestra.run.smithi037.stdout:serviceaccount/rook-csi-rbd-provisioner-sa created 2022-03-10T03:05:57.226 INFO:teuthology.orchestra.run.smithi037.stdout:configmap/rook-ceph-operator-config created 2022-03-10T03:05:57.310 INFO:teuthology.orchestra.run.smithi037.stdout:deployment.apps/rook-ceph-operator created 2022-03-10T03:05:57.315 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-10T03:05:57.395 INFO:teuthology.orchestra.run.smithi037.stdout:NAME READY STATUS RESTARTS AGE 2022-03-10T03:05:57.396 INFO:teuthology.orchestra.run.smithi037.stdout:rook-ceph-operator-75dd789779-lfs4f 0/1 Pending 0 0s 2022-03-10T03:06:07.403 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-10T03:06:07.469 INFO:teuthology.orchestra.run.smithi037.stdout:NAME READY STATUS RESTARTS AGE 2022-03-10T03:06:07.470 INFO:teuthology.orchestra.run.smithi037.stdout:rook-ceph-operator-75dd789779-lfs4f 0/1 Pending 0 10s 2022-03-10T03:06:17.472 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-10T03:06:17.541 INFO:teuthology.orchestra.run.smithi037.stdout:NAME READY STATUS RESTARTS AGE 2022-03-10T03:06:17.542 INFO:teuthology.orchestra.run.smithi037.stdout:rook-ceph-operator-75dd789779-lfs4f 0/1 ContainerCreating 0 20s 2022-03-10T03:06:27.545 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-10T03:06:27.618 INFO:teuthology.orchestra.run.smithi037.stdout:NAME READY STATUS RESTARTS AGE 2022-03-10T03:06:27.618 INFO:teuthology.orchestra.run.smithi037.stdout:rook-ceph-operator-75dd789779-lfs4f 0/1 ContainerCreating 0 30s 2022-03-10T03:06:37.622 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-10T03:06:37.685 INFO:teuthology.orchestra.run.smithi037.stdout:NAME READY STATUS RESTARTS AGE 2022-03-10T03:06:37.686 INFO:teuthology.orchestra.run.smithi037.stdout:rook-ceph-operator-75dd789779-lfs4f 0/1 ContainerCreating 0 40s 2022-03-10T03:06:47.688 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-10T03:06:47.757 INFO:teuthology.orchestra.run.smithi037.stdout:NAME READY STATUS RESTARTS AGE 2022-03-10T03:06:47.757 INFO:teuthology.orchestra.run.smithi037.stdout:rook-ceph-operator-75dd789779-lfs4f 0/1 ContainerCreating 0 50s 2022-03-10T03:06:57.761 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-10T03:06:57.829 INFO:teuthology.orchestra.run.smithi037.stdout:NAME READY STATUS RESTARTS AGE 2022-03-10T03:06:57.830 INFO:teuthology.orchestra.run.smithi037.stdout:rook-ceph-operator-75dd789779-lfs4f 0/1 ContainerCreating 0 60s 2022-03-10T03:07:07.833 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-10T03:07:07.902 INFO:teuthology.orchestra.run.smithi037.stdout:NAME READY STATUS RESTARTS AGE 2022-03-10T03:07:07.903 INFO:teuthology.orchestra.run.smithi037.stdout:rook-ceph-operator-75dd789779-lfs4f 1/1 Running 0 70s 2022-03-10T03:07:07.905 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph logs -f rook-ceph-operator-75dd789779-lfs4f 2022-03-10T03:07:07.914 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2022-03-10T03:07:07.914 DEBUG:teuthology.orchestra.run.smithi037:> dd if=/scratch_devs of=/dev/stdout 2022-03-10T03:07:07.961 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2022-03-10T03:07:07.961 DEBUG:teuthology.orchestra.run.smithi038:> dd if=/scratch_devs of=/dev/stdout 2022-03-10T03:07:07.970 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2022-03-10T03:07:07.970 DEBUG:teuthology.orchestra.run.smithi195:> dd if=/scratch_devs of=/dev/stdout 2022-03-10T03:07:07.981 INFO:tasks.rook: override: [mgr] debug mgr = 20 2022-03-10T03:07:07.981 INFO:tasks.rook: override: [mgr] debug ms = 1 2022-03-10T03:07:07.982 INFO:tasks.rook: override: [mon] debug mon = 20 2022-03-10T03:07:07.982 INFO:tasks.rook: override: [mon] debug ms = 1 2022-03-10T03:07:07.982 INFO:tasks.rook: override: [mon] debug paxos = 20 2022-03-10T03:07:07.983 INFO:tasks.rook: override: [osd] debug ms = 1 2022-03-10T03:07:07.983 INFO:tasks.rook: override: [osd] debug osd = 20 2022-03-10T03:07:07.984 INFO:tasks.rook:Config: b"[global]\n\nlog to file = true\n\nmon clock drift allowed = 1.000\n\n# replicate across OSDs, not hosts\nosd crush chooseleaf type = 0\n\n# enable some debugging\nauth debug = true\nms die on old message = true\nms die on bug = true\ndebug asserts on shutdown = true\n\n\n[osd]\n# debugging\nosd debug shutdown = true\nosd debug op order = true\nosd debug verify stray on activate = true\nosd debug pg log writeout = true\nosd debug verify cached snaps = true\nosd debug verify missing on start = true\nosd debug misdirected ops = true\nosd op queue = debug_random\nosd op queue cut off = debug_random\nosd shutdown pgref assert = true\nbdev debug aio = true\nosd sloppy crc = true\ndebug ms = 1\ndebug osd = 20\n\n\n[mon]\n# rotate auth tickets quickly to exercise renewal paths\nauth mon ticket ttl = 660# 11m\nauth service ticket ttl = 240# 4m\n\n# don't complain about global id reclaim\nmon_warn_on_insecure_global_id_reclaim = false\nmon_warn_on_insecure_global_id_reclaim_allowed = false\ndebug mon = 20\ndebug ms = 1\ndebug paxos = 20\n[mgr]\ndebug mgr = 20\ndebug ms = 1\n" 2022-03-10T03:07:07.986 DEBUG:teuthology.orchestra.run.smithi037:> kubectl create -f - 2022-03-10T03:07:07.998 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:07:04.325580 I | rookcmd: starting Rook v1.8.0-alpha.0.393.ga181988f6 with arguments '/usr/local/bin/rook ceph operator' 2022-03-10T03:07:07.998 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:07:04.325719 I | rookcmd: flag values: --enable-machine-disruption-budget=false, --help=false, --kubeconfig=, --log-level=INFO, --operator-image=, --service-account= 2022-03-10T03:07:07.998 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:07:04.325725 I | cephcmd: starting Rook-Ceph operator 2022-03-10T03:07:07.999 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:07:04.488534 I | cephcmd: base ceph version inside the rook operator image is "ceph version 16.2.7 (dd0603118f56ab514f133c8d2e3adfc983942503) pacific (stable)" 2022-03-10T03:07:07.999 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:07:04.523782 I | op-k8sutil: ROOK_CURRENT_NAMESPACE_ONLY="false" (env var) 2022-03-10T03:07:07.999 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:07:04.523815 I | operator: watching all namespaces for Ceph CRs 2022-03-10T03:07:07.999 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:07:04.523949 I | operator: setting up schemes 2022-03-10T03:07:08.000 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:07:04.528023 I | operator: setting up the controller-runtime manager 2022-03-10T03:07:08.000 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:07:04.832493 I | operator: looking for admission webhook secret "rook-ceph-admission-controller" 2022-03-10T03:07:08.000 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:07:04.835854 I | operator: admission webhook secret "rook-ceph-admission-controller" not found. proceeding without the admission controller 2022-03-10T03:07:08.000 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:07:04.835912 I | ceph-cluster-controller: successfully started 2022-03-10T03:07:08.001 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:07:04.836006 I | ceph-cluster-controller: enabling hotplug orchestration 2022-03-10T03:07:08.001 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:07:04.836049 I | ceph-crashcollector-controller: successfully started 2022-03-10T03:07:08.001 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:07:04.836086 I | ceph-block-pool-controller: successfully started 2022-03-10T03:07:08.001 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:07:04.836148 I | ceph-object-store-user-controller: successfully started 2022-03-10T03:07:08.002 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:07:04.836214 I | ceph-object-realm-controller: successfully started 2022-03-10T03:07:08.002 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:07:04.836242 I | ceph-object-zonegroup-controller: successfully started 2022-03-10T03:07:08.002 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:07:04.836268 I | ceph-object-zone-controller: successfully started 2022-03-10T03:07:08.002 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:07:04.836462 I | ceph-object-controller: successfully started 2022-03-10T03:07:08.003 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:07:04.836524 I | ceph-file-controller: successfully started 2022-03-10T03:07:08.003 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:07:04.836584 I | ceph-nfs-controller: successfully started 2022-03-10T03:07:08.003 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:07:04.836657 I | ceph-rbd-mirror-controller: successfully started 2022-03-10T03:07:08.004 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:07:04.836700 I | ceph-client-controller: successfully started 2022-03-10T03:07:08.004 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:07:04.836743 I | ceph-filesystem-mirror-controller: successfully started 2022-03-10T03:07:08.004 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:07:04.836784 I | operator: rook-ceph-operator-config-controller successfully started 2022-03-10T03:07:08.004 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:07:04.836820 I | ceph-csi: rook-ceph-operator-csi-controller successfully started 2022-03-10T03:07:08.005 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:07:04.836860 I | op-bucket-prov: rook-ceph-operator-bucket-controller successfully started 2022-03-10T03:07:08.005 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:07:04.836901 I | ceph-bucket-topic: successfully started 2022-03-10T03:07:08.005 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:07:04.836929 I | ceph-bucket-notification: successfully started 2022-03-10T03:07:08.005 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:07:04.836952 I | ceph-bucket-notification: successfully started 2022-03-10T03:07:08.006 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:07:04.836980 I | ceph-fs-subvolumegroup-controller: successfully started 2022-03-10T03:07:08.006 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:07:04.838343 I | operator: starting the controller-runtime manager 2022-03-10T03:07:08.006 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:07:04.942940 I | op-k8sutil: ROOK_CEPH_COMMANDS_TIMEOUT_SECONDS="15" (configmap) 2022-03-10T03:07:08.006 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:07:04.942986 I | op-k8sutil: ROOK_LOG_LEVEL="INFO" (configmap) 2022-03-10T03:07:08.007 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:07:04.943007 I | op-k8sutil: ROOK_ENABLE_DISCOVERY_DAEMON="false" (configmap) 2022-03-10T03:07:08.007 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:07:04.945814 I | ceph-csi: CSI Ceph RBD driver disabled 2022-03-10T03:07:08.007 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:07:04.945840 I | op-k8sutil: removing daemonset csi-rbdplugin if it exists 2022-03-10T03:07:08.007 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:07:04.948556 I | operator: rook-ceph-operator-config-controller done reconciling 2022-03-10T03:07:08.008 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:07:04.948797 I | op-k8sutil: removing deployment csi-rbdplugin-provisioner if it exists 2022-03-10T03:07:08.008 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:07:04.956144 I | ceph-csi: successfully removed CSI Ceph RBD driver 2022-03-10T03:07:08.008 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:07:04.956163 I | ceph-csi: CSI CephFS driver disabled 2022-03-10T03:07:08.008 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:07:04.956171 I | op-k8sutil: removing daemonset csi-cephfsplugin if it exists 2022-03-10T03:07:08.008 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:07:04.958097 I | op-k8sutil: removing deployment csi-cephfsplugin-provisioner if it exists 2022-03-10T03:07:08.009 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:07:04.963665 I | ceph-csi: successfully removed CSI CephFS driver 2022-03-10T03:07:08.950 INFO:teuthology.orchestra.run.smithi037.stdout:configmap/rook-config-override created 2022-03-10T03:07:08.955 INFO:tasks.rook:Cluster: apiVersion: ceph.rook.io/v1 kind: CephCluster metadata: name: rook-ceph namespace: rook-ceph spec: cephVersion: allowUnsupported: true image: quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3 dataDirHostPath: /var/lib/rook mgr: count: 1 modules: - enabled: true name: rook mon: allowMultiplePerNode: true count: 3 skipUpgradeChecks: true 2022-03-10T03:07:08.956 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2022-03-10T03:07:08.956 DEBUG:teuthology.orchestra.run.smithi037:> dd of=cluster.yaml 2022-03-10T03:07:08.962 DEBUG:teuthology.orchestra.run.smithi037:> kubectl create -f cluster.yaml 2022-03-10T03:07:11.210 INFO:teuthology.orchestra.run.smithi037.stdout:cephcluster.ceph.rook.io/rook-ceph created 2022-03-10T03:07:11.219 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:07:11.226681 I | ceph-spec: adding finalizer "cephcluster.ceph.rook.io" on "rook-ceph" 2022-03-10T03:07:11.223 DEBUG:teuthology.orchestra.remote:smithi037:rook/deploy/examples/operator.yaml is 20KB 2022-03-10T03:07:11.231 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:07:11.239004 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-03-10T03:07:11.235 DEBUG:teuthology.orchestra.run.smithi037:> kubectl create -f rook/deploy/examples/toolbox.yaml 2022-03-10T03:07:11.239 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:07:11.246980 I | ceph-cluster-controller: clusterInfo not yet found, must be a new cluster. 2022-03-10T03:07:11.251 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:07:11.258331 I | ceph-csi: successfully created csi config map "rook-ceph-csi-config" 2022-03-10T03:07:11.259 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:07:11.266225 I | ceph-spec: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3... 2022-03-10T03:07:11.290 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:07:11.297425 I | op-k8sutil: ROOK_CSI_ENABLE_RBD="true" (configmap) 2022-03-10T03:07:11.291 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:07:11.297447 I | op-k8sutil: ROOK_CSI_ENABLE_CEPHFS="true" (configmap) 2022-03-10T03:07:11.291 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:07:11.297454 I | op-k8sutil: ROOK_CSI_ALLOW_UNSUPPORTED_VERSION="false" (configmap) 2022-03-10T03:07:11.291 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:07:11.297462 I | op-k8sutil: ROOK_CSI_ENABLE_GRPC_METRICS="false" (configmap) 2022-03-10T03:07:11.291 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:07:11.297471 I | op-k8sutil: CSI_ENABLE_HOST_NETWORK="true" (default) 2022-03-10T03:07:11.292 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:07:11.297481 I | op-k8sutil: ROOK_CSI_CEPH_IMAGE="quay.io/cephcsi/cephcsi:v3.5.1" (default) 2022-03-10T03:07:11.292 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:07:11.297488 I | op-k8sutil: ROOK_CSI_REGISTRAR_IMAGE="k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.5.0" (default) 2022-03-10T03:07:11.292 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:07:11.297495 I | op-k8sutil: ROOK_CSI_PROVISIONER_IMAGE="k8s.gcr.io/sig-storage/csi-provisioner:v3.1.0" (default) 2022-03-10T03:07:11.292 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:07:11.297502 I | op-k8sutil: ROOK_CSI_ATTACHER_IMAGE="k8s.gcr.io/sig-storage/csi-attacher:v3.4.0" (default) 2022-03-10T03:07:11.293 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:07:11.297508 I | op-k8sutil: ROOK_CSI_SNAPSHOTTER_IMAGE="k8s.gcr.io/sig-storage/csi-snapshotter:v5.0.1" (default) 2022-03-10T03:07:11.293 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:07:11.297514 I | op-k8sutil: ROOK_CSI_KUBELET_DIR_PATH="/var/lib/kubelet" (default) 2022-03-10T03:07:11.293 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:07:11.297521 I | op-k8sutil: CSI_VOLUME_REPLICATION_IMAGE="quay.io/csiaddons/volumereplication-operator:v0.3.0" (default) 2022-03-10T03:07:11.294 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:07:11.297528 I | op-k8sutil: ROOK_CSIADDONS_IMAGE="quay.io/csiaddons/k8s-sidecar:v0.2.1" (default) 2022-03-10T03:07:11.294 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:07:11.297535 I | op-k8sutil: ROOK_CSI_CEPHFS_POD_LABELS="" (default) 2022-03-10T03:07:11.294 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:07:11.297544 I | op-k8sutil: ROOK_CSI_RBD_POD_LABELS="" (default) 2022-03-10T03:07:11.294 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:07:11.297555 I | ceph-csi: detecting the ceph csi image version for image "quay.io/cephcsi/cephcsi:v3.5.1" 2022-03-10T03:07:11.300 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:07:11.297623 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-03-10T03:07:11.301 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:07:11.297646 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-10T03:07:11.545 INFO:teuthology.orchestra.run.smithi037.stdout:deployment.apps/rook-ceph-tools created 2022-03-10T03:07:11.552 INFO:tasks.rook:Waiting for tools container to start 2022-03-10T03:07:11.553 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-10T03:07:11.717 INFO:teuthology.orchestra.run.smithi037.stdout:NAME READY STATUS RESTARTS AGE 2022-03-10T03:07:11.717 INFO:teuthology.orchestra.run.smithi037.stdout:rook-ceph-tools-d6d7c985c-sj4qk 0/1 Pending 0 0s 2022-03-10T03:07:16.721 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-10T03:07:16.787 INFO:teuthology.orchestra.run.smithi037.stdout:NAME READY STATUS RESTARTS AGE 2022-03-10T03:07:16.788 INFO:teuthology.orchestra.run.smithi037.stdout:rook-ceph-tools-d6d7c985c-sj4qk 0/1 ContainerCreating 0 5s 2022-03-10T03:07:21.792 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-10T03:07:21.881 INFO:teuthology.orchestra.run.smithi037.stdout:NAME READY STATUS RESTARTS AGE 2022-03-10T03:07:21.881 INFO:teuthology.orchestra.run.smithi037.stdout:rook-ceph-tools-d6d7c985c-sj4qk 0/1 ContainerCreating 0 10s 2022-03-10T03:07:26.883 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-10T03:07:26.953 INFO:teuthology.orchestra.run.smithi037.stdout:NAME READY STATUS RESTARTS AGE 2022-03-10T03:07:26.954 INFO:teuthology.orchestra.run.smithi037.stdout:rook-ceph-tools-d6d7c985c-sj4qk 0/1 ContainerCreating 0 15s 2022-03-10T03:07:31.956 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-10T03:07:32.020 INFO:teuthology.orchestra.run.smithi037.stdout:NAME READY STATUS RESTARTS AGE 2022-03-10T03:07:32.020 INFO:teuthology.orchestra.run.smithi037.stdout:rook-ceph-tools-d6d7c985c-sj4qk 0/1 ContainerCreating 0 21s 2022-03-10T03:07:37.023 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-10T03:07:37.091 INFO:teuthology.orchestra.run.smithi037.stdout:NAME READY STATUS RESTARTS AGE 2022-03-10T03:07:37.091 INFO:teuthology.orchestra.run.smithi037.stdout:rook-ceph-tools-d6d7c985c-sj4qk 0/1 ContainerCreating 0 26s 2022-03-10T03:07:42.095 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-10T03:07:42.163 INFO:teuthology.orchestra.run.smithi037.stdout:NAME READY STATUS RESTARTS AGE 2022-03-10T03:07:42.163 INFO:teuthology.orchestra.run.smithi037.stdout:rook-ceph-tools-d6d7c985c-sj4qk 0/1 ContainerCreating 0 31s 2022-03-10T03:07:47.166 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-10T03:07:47.242 INFO:teuthology.orchestra.run.smithi037.stdout:NAME READY STATUS RESTARTS AGE 2022-03-10T03:07:47.243 INFO:teuthology.orchestra.run.smithi037.stdout:rook-ceph-tools-d6d7c985c-sj4qk 0/1 ContainerCreating 0 36s 2022-03-10T03:07:52.254 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-10T03:07:52.325 INFO:teuthology.orchestra.run.smithi037.stdout:NAME READY STATUS RESTARTS AGE 2022-03-10T03:07:52.325 INFO:teuthology.orchestra.run.smithi037.stdout:rook-ceph-tools-d6d7c985c-sj4qk 0/1 ContainerCreating 0 41s 2022-03-10T03:07:57.329 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-10T03:07:57.406 INFO:teuthology.orchestra.run.smithi037.stdout:NAME READY STATUS RESTARTS AGE 2022-03-10T03:07:57.407 INFO:teuthology.orchestra.run.smithi037.stdout:rook-ceph-tools-d6d7c985c-sj4qk 0/1 ContainerCreating 0 46s 2022-03-10T03:08:02.410 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-10T03:08:02.473 INFO:teuthology.orchestra.run.smithi037.stdout:NAME READY STATUS RESTARTS AGE 2022-03-10T03:08:02.474 INFO:teuthology.orchestra.run.smithi037.stdout:rook-ceph-tools-d6d7c985c-sj4qk 0/1 ContainerCreating 0 51s 2022-03-10T03:08:07.477 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-10T03:08:07.557 INFO:teuthology.orchestra.run.smithi037.stdout:NAME READY STATUS RESTARTS AGE 2022-03-10T03:08:07.558 INFO:teuthology.orchestra.run.smithi037.stdout:rook-ceph-tools-d6d7c985c-sj4qk 0/1 ContainerCreating 0 56s 2022-03-10T03:08:12.560 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-10T03:08:12.643 INFO:teuthology.orchestra.run.smithi037.stdout:NAME READY STATUS RESTARTS AGE 2022-03-10T03:08:12.643 INFO:teuthology.orchestra.run.smithi037.stdout:rook-ceph-tools-d6d7c985c-sj4qk 0/1 ContainerCreating 0 61s 2022-03-10T03:08:17.646 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-10T03:08:17.713 INFO:teuthology.orchestra.run.smithi037.stdout:NAME READY STATUS RESTARTS AGE 2022-03-10T03:08:17.713 INFO:teuthology.orchestra.run.smithi037.stdout:rook-ceph-tools-d6d7c985c-sj4qk 0/1 ContainerCreating 0 66s 2022-03-10T03:08:22.716 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-10T03:08:22.778 INFO:teuthology.orchestra.run.smithi037.stdout:NAME READY STATUS RESTARTS AGE 2022-03-10T03:08:22.779 INFO:teuthology.orchestra.run.smithi037.stdout:rook-ceph-tools-d6d7c985c-sj4qk 0/1 ContainerCreating 0 71s 2022-03-10T03:08:27.782 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-10T03:08:27.849 INFO:teuthology.orchestra.run.smithi037.stdout:NAME READY STATUS RESTARTS AGE 2022-03-10T03:08:27.849 INFO:teuthology.orchestra.run.smithi037.stdout:rook-ceph-tools-d6d7c985c-sj4qk 0/1 ContainerCreating 0 76s 2022-03-10T03:08:32.852 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-10T03:08:32.926 INFO:teuthology.orchestra.run.smithi037.stdout:NAME READY STATUS RESTARTS AGE 2022-03-10T03:08:32.926 INFO:teuthology.orchestra.run.smithi037.stdout:rook-ceph-tools-d6d7c985c-sj4qk 0/1 ContainerCreating 0 81s 2022-03-10T03:08:37.929 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-10T03:08:37.996 INFO:teuthology.orchestra.run.smithi037.stdout:NAME READY STATUS RESTARTS AGE 2022-03-10T03:08:37.996 INFO:teuthology.orchestra.run.smithi037.stdout:rook-ceph-tools-d6d7c985c-sj4qk 0/1 ContainerCreating 0 87s 2022-03-10T03:08:42.998 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-10T03:08:43.069 INFO:teuthology.orchestra.run.smithi037.stdout:NAME READY STATUS RESTARTS AGE 2022-03-10T03:08:43.069 INFO:teuthology.orchestra.run.smithi037.stdout:rook-ceph-tools-d6d7c985c-sj4qk 0/1 ContainerCreating 0 92s 2022-03-10T03:08:47.972 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:08:47.977750 I | ceph-csi: Detected ceph CSI image version: "v3.5.1" 2022-03-10T03:08:47.972 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:08:47.977791 I | op-k8sutil: CSI_FORCE_CEPHFS_KERNEL_CLIENT="true" (configmap) 2022-03-10T03:08:47.973 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:08:47.977807 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default) 2022-03-10T03:08:47.973 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:08:47.977817 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default) 2022-03-10T03:08:47.973 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:08:47.977828 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default) 2022-03-10T03:08:47.973 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:08:47.977840 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default) 2022-03-10T03:08:47.973 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:08:47.977853 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default) 2022-03-10T03:08:47.974 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:08:47.977865 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default) 2022-03-10T03:08:47.974 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:08:47.977879 I | op-k8sutil: CSIADDONS_PORT="9070" (default) 2022-03-10T03:08:47.974 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:08:47.977891 I | op-k8sutil: CSIADDONS_PORT="9070" (default) 2022-03-10T03:08:47.974 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:08:47.977905 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default) 2022-03-10T03:08:47.975 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:08:47.977917 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default) 2022-03-10T03:08:47.975 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:08:47.977928 I | op-k8sutil: CSI_PLUGIN_PRIORITY_CLASSNAME="" (default) 2022-03-10T03:08:47.975 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:08:47.977941 I | op-k8sutil: CSI_PROVISIONER_PRIORITY_CLASSNAME="" (default) 2022-03-10T03:08:47.975 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:08:47.977954 I | op-k8sutil: CSI_ENABLE_OMAP_GENERATOR="false" (default) 2022-03-10T03:08:47.975 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:08:47.977967 I | op-k8sutil: CSI_ENABLE_RBD_SNAPSHOTTER="true" (configmap) 2022-03-10T03:08:47.976 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:08:47.977979 I | op-k8sutil: CSI_ENABLE_CEPHFS_SNAPSHOTTER="true" (configmap) 2022-03-10T03:08:47.976 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:08:47.977992 I | op-k8sutil: CSI_ENABLE_VOLUME_REPLICATION="false" (configmap) 2022-03-10T03:08:47.976 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:08:47.978005 I | op-k8sutil: CSI_ENABLE_CSIADDONS="false" (configmap) 2022-03-10T03:08:47.976 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:08:47.978019 I | op-k8sutil: CSI_CEPHFS_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-03-10T03:08:47.977 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:08:47.978032 I | op-k8sutil: CSI_RBD_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-03-10T03:08:47.977 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:08:47.978045 I | op-k8sutil: CSI_PLUGIN_ENABLE_SELINUX_HOST_MOUNT="false" (configmap) 2022-03-10T03:08:47.977 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:08:47.978057 I | ceph-csi: Kubernetes version is 1.23 2022-03-10T03:08:47.978 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:08:47.978073 I | op-k8sutil: ROOK_CSI_RESIZER_IMAGE="k8s.gcr.io/sig-storage/csi-resizer:v1.4.0" (default) 2022-03-10T03:08:47.978 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:08:47.978104 I | op-k8sutil: CSI_LOG_LEVEL="" (default) 2022-03-10T03:08:47.996 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:08:48.003773 I | op-k8sutil: CSI_PROVISIONER_REPLICAS="2" (configmap) 2022-03-10T03:08:48.007 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:08:48.015376 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-03-10T03:08:48.007 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:08:48.015409 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-10T03:08:48.008 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:08:48.015414 I | op-k8sutil: CSI_PLUGIN_TOLERATIONS="" (default) 2022-03-10T03:08:48.008 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:08:48.015417 I | op-k8sutil: CSI_PLUGIN_NODE_AFFINITY="" (default) 2022-03-10T03:08:48.008 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:08:48.015420 I | op-k8sutil: CSI_RBD_PLUGIN_TOLERATIONS="" (default) 2022-03-10T03:08:48.008 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:08:48.015423 I | op-k8sutil: CSI_RBD_PLUGIN_NODE_AFFINITY="" (default) 2022-03-10T03:08:48.009 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:08:48.015427 I | op-k8sutil: CSI_RBD_PLUGIN_RESOURCE="" (default) 2022-03-10T03:08:48.064 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:08:48.072429 I | op-k8sutil: CSI_RBD_PROVISIONER_TOLERATIONS="" (default) 2022-03-10T03:08:48.065 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:08:48.072464 I | op-k8sutil: CSI_RBD_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-10T03:08:48.065 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:08:48.072474 I | op-k8sutil: CSI_RBD_PROVISIONER_RESOURCE="" (default) 2022-03-10T03:08:48.072 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-10T03:08:48.139 INFO:teuthology.orchestra.run.smithi037.stdout:NAME READY STATUS RESTARTS AGE 2022-03-10T03:08:48.139 INFO:teuthology.orchestra.run.smithi037.stdout:rook-ceph-tools-d6d7c985c-sj4qk 0/1 ContainerCreating 0 97s 2022-03-10T03:08:48.168 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:08:48.176499 I | ceph-csi: successfully started CSI Ceph RBD driver 2022-03-10T03:08:48.289 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:08:48.296361 I | op-k8sutil: CSI_CEPHFS_PLUGIN_TOLERATIONS="" (default) 2022-03-10T03:08:48.289 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:08:48.296387 I | op-k8sutil: CSI_CEPHFS_PLUGIN_NODE_AFFINITY="" (default) 2022-03-10T03:08:48.290 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:08:48.296394 I | op-k8sutil: CSI_CEPHFS_PLUGIN_RESOURCE="" (default) 2022-03-10T03:08:48.335 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:08:48.344044 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_TOLERATIONS="" (default) 2022-03-10T03:08:48.336 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:08:48.344071 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-10T03:08:48.336 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:08:48.344076 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_RESOURCE="" (default) 2022-03-10T03:08:48.451 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:08:48.458208 I | ceph-csi: successfully started CSI CephFS driver 2022-03-10T03:08:48.621 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:08:48.629008 I | op-k8sutil: CSI_RBD_FSGROUPPOLICY="ReadWriteOnceWithFSType" (configmap) 2022-03-10T03:08:48.644 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:08:48.651428 I | ceph-csi: CSIDriver object created for driver "rook-ceph.rbd.csi.ceph.com" 2022-03-10T03:08:48.644 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:08:48.651458 I | op-k8sutil: CSI_CEPHFS_FSGROUPPOLICY="ReadWriteOnceWithFSType" (configmap) 2022-03-10T03:08:48.689 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:08:48.697947 I | ceph-csi: CSIDriver object created for driver "rook-ceph.cephfs.csi.ceph.com" 2022-03-10T03:08:53.166 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-10T03:08:53.286 INFO:teuthology.orchestra.run.smithi037.stdout:NAME READY STATUS RESTARTS AGE 2022-03-10T03:08:53.287 INFO:teuthology.orchestra.run.smithi037.stdout:rook-ceph-tools-d6d7c985c-sj4qk 0/1 ContainerCreating 0 102s 2022-03-10T03:08:57.660 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:08:57.668012 I | ceph-spec: detected ceph image version: "17.0.0-11025 quincy" 2022-03-10T03:08:57.661 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:08:57.668041 I | ceph-cluster-controller: validating ceph version from provided image 2022-03-10T03:08:57.661 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:08:57.668060 W | ceph-cluster-controller: unsupported ceph version detected: "17.0.0-11025 quincy", pursuing 2022-03-10T03:08:57.783 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:08:57.791146 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-11025 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3" 2022-03-10T03:08:58.004 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:08:58.012943 E | ceph-spec: failed to update cluster condition to {Type:Progressing Status:True Reason:ClusterProgressing Message:Configuring the Ceph cluster LastHeartbeatTime:2022-03-10 03:08:57.987782686 +0000 UTC m=+113.681616762 LastTransitionTime:2022-03-10 03:08:57.987782451 +0000 UTC m=+113.681616573}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-10T03:08:58.031 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:08:58.040222 I | op-mon: start running mons 2022-03-10T03:08:58.141 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:08:58.149838 I | op-mon: creating mon secrets for a new cluster 2022-03-10T03:08:58.165 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:08:58.174078 I | op-mon: existing maxMonID not found or failed to load. configmaps "rook-ceph-mon-endpoints" not found 2022-03-10T03:08:58.170 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:08:58.177074 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":[]}] data: mapping:{"node":{}} maxMonId:-1] 2022-03-10T03:08:58.290 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-10T03:08:58.386 INFO:teuthology.orchestra.run.smithi037.stdout:NAME READY STATUS RESTARTS AGE 2022-03-10T03:08:58.386 INFO:teuthology.orchestra.run.smithi037.stdout:rook-ceph-tools-d6d7c985c-sj4qk 0/1 ContainerCreating 0 107s 2022-03-10T03:08:58.409 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:08:58.417828 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-10T03:08:58.409 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:08:58.418053 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-10T03:09:00.010 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:09:00.019293 I | op-mon: targeting the mon count 3 2022-03-10T03:09:00.058 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:09:00.067060 I | op-mon: created canary deployment rook-ceph-mon-a-canary 2022-03-10T03:09:00.066 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:09:00.074760 I | op-mon: created canary deployment rook-ceph-mon-b-canary 2022-03-10T03:09:00.094 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:09:00.100162 I | op-mon: created canary deployment rook-ceph-mon-c-canary 2022-03-10T03:09:00.893 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:09:00.901667 I | op-mon: canary monitor deployment rook-ceph-mon-a-canary scheduled to smithi038 2022-03-10T03:09:00.893 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:09:00.901701 I | op-mon: mon a assigned to node smithi038 2022-03-10T03:09:01.009 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:09:01.017994 I | op-mon: canary monitor deployment rook-ceph-mon-b-canary scheduled to smithi195 2022-03-10T03:09:01.009 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:09:01.018020 I | op-mon: mon b assigned to node smithi195 2022-03-10T03:09:01.209 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:09:01.217845 I | op-mon: canary monitor deployment rook-ceph-mon-c-canary scheduled to smithi037 2022-03-10T03:09:01.210 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:09:01.217866 I | op-mon: mon c assigned to node smithi037 2022-03-10T03:09:01.214 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:09:01.223501 I | op-mon: cleaning up canary monitor deployment "rook-ceph-mon-a-canary" 2022-03-10T03:09:01.219 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:09:01.228447 I | op-mon: cleaning up canary monitor deployment "rook-ceph-mon-b-canary" 2022-03-10T03:09:01.422 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:09:01.430655 I | op-mon: cleaning up canary monitor deployment "rook-ceph-mon-c-canary" 2022-03-10T03:09:01.544 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:09:01.553061 I | op-mon: creating mon a 2022-03-10T03:09:01.724 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:09:01.731899 I | op-mon: parsing mon endpoints: 2022-03-10T03:09:01.724 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:09:01.731937 W | op-mon: ignoring invalid monitor 2022-03-10T03:09:01.725 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:09:01.731971 I | op-k8sutil: ROOK_OBC_WATCH_OPERATOR_NAMESPACE="true" (configmap) 2022-03-10T03:09:01.725 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:09:01.731980 I | op-bucket-prov: ceph bucket provisioner launched watching for provisioner "rook-ceph.ceph.rook.io/bucket" 2022-03-10T03:09:01.726 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:09:01.734049 I | op-bucket-prov: successfully reconciled bucket provisioner 2022-03-10T03:09:01.727 INFO:tasks.rook.operator.smithi037.stdout:I0310 03:09:01.734145 1 manager.go:135] objectbucket.io/provisioner-manager "msg"="starting provisioner" "name"="rook-ceph.ceph.rook.io/bucket" 2022-03-10T03:09:01.983 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:09:01.992041 I | op-mon: mon "a" endpoint is [v2:10.100.186.100:3300,v1:10.100.186.100:6789] 2022-03-10T03:09:02.409 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:09:02.417751 I | op-mon: monitor endpoints changed, updating the bootstrap peer token 2022-03-10T03:09:02.410 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:09:02.417851 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.100.186.100:6789"]}] data:a=10.100.186.100:6789 mapping:{"node":{"a":{"Name":"smithi038","Hostname":"smithi038","Address":"172.21.15.38"},"b":{"Name":"smithi195","Hostname":"smithi195","Address":"172.21.15.195"},"c":{"Name":"smithi037","Hostname":"smithi037","Address":"172.21.15.37"}}} maxMonId:-1] 2022-03-10T03:09:02.410 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:09:02.417879 I | op-mon: monitor endpoints changed, updating the bootstrap peer token 2022-03-10T03:09:03.008 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:09:03.017274 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-10T03:09:03.011 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:09:03.017572 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-10T03:09:03.389 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-10T03:09:03.421 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:09:03.430266 I | op-mon: 0 of 1 expected mons are ready. creating or updating deployments without checking quorum in attempt to achieve a healthy mon cluster 2022-03-10T03:09:03.457 INFO:teuthology.orchestra.run.smithi037.stdout:NAME READY STATUS RESTARTS AGE 2022-03-10T03:09:03.457 INFO:teuthology.orchestra.run.smithi037.stdout:rook-ceph-tools-d6d7c985c-sj4qk 0/1 ContainerCreating 0 112s 2022-03-10T03:09:03.638 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:09:03.646282 I | op-mon: updating maxMonID from -1 to 0 after committing mon "a" 2022-03-10T03:09:04.409 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:09:04.417495 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.100.186.100:6789"]}] data:a=10.100.186.100:6789 mapping:{"node":{"a":{"Name":"smithi038","Hostname":"smithi038","Address":"172.21.15.38"},"b":{"Name":"smithi195","Hostname":"smithi195","Address":"172.21.15.195"},"c":{"Name":"smithi037","Hostname":"smithi037","Address":"172.21.15.37"}}} maxMonId:0] 2022-03-10T03:09:04.409 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:09:04.417527 I | op-mon: waiting for mon quorum with [a] 2022-03-10T03:09:04.615 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:09:04.623550 I | op-mon: mons running: [a] 2022-03-10T03:09:08.460 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-10T03:09:08.534 INFO:teuthology.orchestra.run.smithi037.stdout:NAME READY STATUS RESTARTS AGE 2022-03-10T03:09:08.534 INFO:teuthology.orchestra.run.smithi037.stdout:rook-ceph-tools-d6d7c985c-sj4qk 0/1 ContainerCreating 0 117s 2022-03-10T03:09:13.538 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-10T03:09:13.626 INFO:teuthology.orchestra.run.smithi037.stdout:NAME READY STATUS RESTARTS AGE 2022-03-10T03:09:13.626 INFO:teuthology.orchestra.run.smithi037.stdout:rook-ceph-tools-d6d7c985c-sj4qk 0/1 ContainerCreating 0 2m2s 2022-03-10T03:09:18.630 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-10T03:09:18.717 INFO:teuthology.orchestra.run.smithi037.stdout:NAME READY STATUS RESTARTS AGE 2022-03-10T03:09:18.718 INFO:teuthology.orchestra.run.smithi037.stdout:rook-ceph-tools-d6d7c985c-sj4qk 0/1 ContainerCreating 0 2m7s 2022-03-10T03:09:23.720 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-10T03:09:23.790 INFO:teuthology.orchestra.run.smithi037.stdout:NAME READY STATUS RESTARTS AGE 2022-03-10T03:09:23.790 INFO:teuthology.orchestra.run.smithi037.stdout:rook-ceph-tools-d6d7c985c-sj4qk 0/1 ContainerCreating 0 2m12s 2022-03-10T03:09:24.767 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:09:24.775825 I | op-mon: mons running: [a] 2022-03-10T03:09:28.793 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-10T03:09:28.873 INFO:teuthology.orchestra.run.smithi037.stdout:NAME READY STATUS RESTARTS AGE 2022-03-10T03:09:28.874 INFO:teuthology.orchestra.run.smithi037.stdout:rook-ceph-tools-d6d7c985c-sj4qk 0/1 ContainerCreating 0 2m17s 2022-03-10T03:09:33.876 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-10T03:09:34.157 INFO:teuthology.orchestra.run.smithi037.stdout:NAME READY STATUS RESTARTS AGE 2022-03-10T03:09:34.157 INFO:teuthology.orchestra.run.smithi037.stdout:rook-ceph-tools-d6d7c985c-sj4qk 0/1 ContainerCreating 0 2m23s 2022-03-10T03:09:39.159 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-10T03:09:39.237 INFO:teuthology.orchestra.run.smithi037.stdout:NAME READY STATUS RESTARTS AGE 2022-03-10T03:09:39.238 INFO:teuthology.orchestra.run.smithi037.stdout:rook-ceph-tools-d6d7c985c-sj4qk 1/1 Running 0 2m28s 2022-03-10T03:09:39.239 INFO:tasks.rook:Waiting for mgr/rook orchestrator to be available 2022-03-10T03:09:39.240 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph orch status -f json 2022-03-10T03:09:45.247 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:09:45.255690 I | op-mon: mons running: [a] 2022-03-10T03:10:05.747 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:10:05.754547 I | op-mon: mon a is not yet running 2022-03-10T03:10:05.748 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:10:05.754569 I | op-mon: mons running: [] 2022-03-10T03:10:26.133 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:10:26.142567 I | op-mon: mon a is not yet running 2022-03-10T03:10:26.134 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:10:26.142606 I | op-mon: mons running: [] 2022-03-10T03:10:45.003 INFO:teuthology.orchestra.run.smithi037.stdout: 2022-03-10T03:10:45.004 INFO:teuthology.orchestra.run.smithi037.stderr:no valid command found; 10 closest matches: 2022-03-10T03:10:45.004 INFO:teuthology.orchestra.run.smithi037.stderr:pg stat 2022-03-10T03:10:45.005 INFO:teuthology.orchestra.run.smithi037.stderr:pg getmap 2022-03-10T03:10:45.005 INFO:teuthology.orchestra.run.smithi037.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-10T03:10:45.005 INFO:teuthology.orchestra.run.smithi037.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-10T03:10:45.005 INFO:teuthology.orchestra.run.smithi037.stderr:pg dump_pools_json 2022-03-10T03:10:45.006 INFO:teuthology.orchestra.run.smithi037.stderr:pg ls-by-pool [...] 2022-03-10T03:10:45.006 INFO:teuthology.orchestra.run.smithi037.stderr:pg ls-by-primary [] [...] 2022-03-10T03:10:45.006 INFO:teuthology.orchestra.run.smithi037.stderr:pg ls-by-osd [] [...] 2022-03-10T03:10:45.006 INFO:teuthology.orchestra.run.smithi037.stderr:pg ls [] [...] 2022-03-10T03:10:45.007 INFO:teuthology.orchestra.run.smithi037.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-10T03:10:45.007 INFO:teuthology.orchestra.run.smithi037.stderr:Error EINVAL: invalid command 2022-03-10T03:10:45.009 INFO:teuthology.orchestra.run.smithi037.stderr:command terminated with exit code 22 2022-03-10T03:10:45.012 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-10T03:10:46.267 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:10:46.276726 I | op-mon: mons running: [a] 2022-03-10T03:10:46.525 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:10:46.535493 I | op-mon: Monitors in quorum: [a] 2022-03-10T03:10:46.526 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:10:46.535529 I | op-mon: mons created: 1 2022-03-10T03:10:46.768 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:10:46.778073 I | op-mon: waiting for mon quorum with [a] 2022-03-10T03:10:46.777 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:10:46.787125 I | op-mon: mons running: [a] 2022-03-10T03:10:46.999 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:10:47.008915 I | op-mon: Monitors in quorum: [a] 2022-03-10T03:10:46.999 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:10:47.008960 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-10T03:10:47.274 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:10:47.283696 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-10T03:10:47.274 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:10:47.283728 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-03-10T03:10:47.576 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:10:47.586670 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-03-10T03:10:47.577 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:10:47.586701 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-10T03:10:47.874 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:10:47.884267 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-10T03:10:47.875 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:10:47.884298 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-03-10T03:10:48.177 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:10:48.187199 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-03-10T03:10:48.177 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:10:48.187227 I | op-config: deleting "log file" option from the mon configuration database 2022-03-10T03:10:48.404 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:10:48.413948 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-03-10T03:10:48.404 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:10:48.413978 I | op-mon: creating mon b 2022-03-10T03:10:48.528 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:10:48.537888 I | op-mon: mon "a" endpoint is [v2:10.100.186.100:3300,v1:10.100.186.100:6789] 2022-03-10T03:10:48.557 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:10:48.567385 I | op-mon: mon "b" endpoint is [v2:10.111.132.151:3300,v1:10.111.132.151:6789] 2022-03-10T03:10:48.649 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:10:48.659160 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.100.186.100:6789","10.111.132.151:6789"]}] data:a=10.100.186.100:6789,b=10.111.132.151:6789 mapping:{"node":{"a":{"Name":"smithi038","Hostname":"smithi038","Address":"172.21.15.38"},"b":{"Name":"smithi195","Hostname":"smithi195","Address":"172.21.15.195"},"c":{"Name":"smithi037","Hostname":"smithi037","Address":"172.21.15.37"}}} maxMonId:0] 2022-03-10T03:10:48.688 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:10:48.698046 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-10T03:10:48.688 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:10:48.698267 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-10T03:10:49.038 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:10:49.048504 I | op-mon: deployment for mon rook-ceph-mon-a already exists. updating if needed 2022-03-10T03:10:49.049 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:10:49.059892 I | op-k8sutil: deployment "rook-ceph-mon-a" did not change, nothing to update 2022-03-10T03:10:49.050 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:10:49.059914 I | op-mon: waiting for mon quorum with [a b] 2022-03-10T03:10:49.412 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:10:49.422085 I | op-mon: mon b is not yet running 2022-03-10T03:10:49.412 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:10:49.422120 I | op-mon: mons running: [a] 2022-03-10T03:10:49.634 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:10:49.643955 I | op-mon: Monitors in quorum: [a] 2022-03-10T03:10:49.716 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:10:49.726348 I | op-mon: updating maxMonID from 0 to 1 after committing mon "b" 2022-03-10T03:10:50.411 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:10:50.420691 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.100.186.100:6789","10.111.132.151:6789"]}] data:a=10.100.186.100:6789,b=10.111.132.151:6789 mapping:{"node":{"a":{"Name":"smithi038","Hostname":"smithi038","Address":"172.21.15.38"},"b":{"Name":"smithi195","Hostname":"smithi195","Address":"172.21.15.195"},"c":{"Name":"smithi037","Hostname":"smithi037","Address":"172.21.15.37"}}} maxMonId:1] 2022-03-10T03:10:50.411 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:10:50.420731 I | op-mon: waiting for mon quorum with [a b] 2022-03-10T03:10:50.813 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:10:50.823211 I | op-mon: mon b is not yet running 2022-03-10T03:10:50.813 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:10:50.823248 I | op-mon: mons running: [a] 2022-03-10T03:10:51.042 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:10:51.051713 I | op-mon: Monitors in quorum: [a] 2022-03-10T03:10:51.042 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:10:51.051746 I | op-mon: mons created: 2 2022-03-10T03:10:51.281 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:10:51.291219 I | op-mon: waiting for mon quorum with [a b] 2022-03-10T03:10:51.298 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:10:51.308217 I | op-mon: mon b is not yet running 2022-03-10T03:10:51.298 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:10:51.308250 I | op-mon: mons running: [a] 2022-03-10T03:10:55.013 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph orch status -f json 2022-03-10T03:10:55.546 INFO:teuthology.orchestra.run.smithi037.stdout: 2022-03-10T03:10:55.548 INFO:teuthology.orchestra.run.smithi037.stderr:no valid command found; 10 closest matches: 2022-03-10T03:10:55.548 INFO:teuthology.orchestra.run.smithi037.stderr:pg stat 2022-03-10T03:10:55.548 INFO:teuthology.orchestra.run.smithi037.stderr:pg getmap 2022-03-10T03:10:55.548 INFO:teuthology.orchestra.run.smithi037.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-10T03:10:55.548 INFO:teuthology.orchestra.run.smithi037.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-10T03:10:55.549 INFO:teuthology.orchestra.run.smithi037.stderr:pg dump_pools_json 2022-03-10T03:10:55.549 INFO:teuthology.orchestra.run.smithi037.stderr:pg ls-by-pool [...] 2022-03-10T03:10:55.549 INFO:teuthology.orchestra.run.smithi037.stderr:pg ls-by-primary [] [...] 2022-03-10T03:10:55.549 INFO:teuthology.orchestra.run.smithi037.stderr:pg ls-by-osd [] [...] 2022-03-10T03:10:55.549 INFO:teuthology.orchestra.run.smithi037.stderr:pg ls [] [...] 2022-03-10T03:10:55.550 INFO:teuthology.orchestra.run.smithi037.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-10T03:10:55.550 INFO:teuthology.orchestra.run.smithi037.stderr:Error EINVAL: invalid command 2022-03-10T03:10:55.550 INFO:teuthology.orchestra.run.smithi037.stderr:command terminated with exit code 22 2022-03-10T03:10:55.552 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-10T03:10:56.316 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:10:56.326858 I | op-mon: mon b is not yet running 2022-03-10T03:10:56.317 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:10:56.326906 I | op-mon: mons running: [a] 2022-03-10T03:11:01.330 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:11:01.340588 I | op-mon: mons running: [a b] 2022-03-10T03:11:02.981 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:11:02.991366 I | op-mon: Monitors in quorum: [a b] 2022-03-10T03:11:02.982 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:11:02.991420 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-10T03:11:03.196 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:11:03.206481 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-10T03:11:03.196 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:11:03.206517 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-03-10T03:11:03.413 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:11:03.423015 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-03-10T03:11:03.413 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:11:03.423042 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-10T03:11:03.651 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:11:03.660966 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-10T03:11:03.651 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:11:03.660994 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-03-10T03:11:03.892 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:11:03.902574 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-03-10T03:11:03.893 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:11:03.902623 I | op-config: deleting "log file" option from the mon configuration database 2022-03-10T03:11:04.115 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:11:04.125222 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-03-10T03:11:04.115 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:11:04.125252 I | op-mon: creating mon c 2022-03-10T03:11:04.270 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:11:04.280572 I | op-mon: mon "a" endpoint is [v2:10.100.186.100:3300,v1:10.100.186.100:6789] 2022-03-10T03:11:04.412 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:11:04.422136 I | op-mon: mon "b" endpoint is [v2:10.111.132.151:3300,v1:10.111.132.151:6789] 2022-03-10T03:11:04.442 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:11:04.452160 I | op-mon: mon "c" endpoint is [v2:10.101.46.100:3300,v1:10.101.46.100:6789] 2022-03-10T03:11:04.546 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:11:04.556761 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.101.46.100:6789","10.100.186.100:6789","10.111.132.151:6789"]}] data:c=10.101.46.100:6789,a=10.100.186.100:6789,b=10.111.132.151:6789 mapping:{"node":{"a":{"Name":"smithi038","Hostname":"smithi038","Address":"172.21.15.38"},"b":{"Name":"smithi195","Hostname":"smithi195","Address":"172.21.15.195"},"c":{"Name":"smithi037","Hostname":"smithi037","Address":"172.21.15.37"}}} maxMonId:1] 2022-03-10T03:11:05.120 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:11:05.130120 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-10T03:11:05.124 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:11:05.130429 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-10T03:11:05.553 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph orch status -f json 2022-03-10T03:11:05.555 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:11:05.565130 I | op-mon: 2 of 3 expected mon deployments exist. creating new deployment(s). 2022-03-10T03:11:05.561 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:11:05.572385 I | op-mon: deployment for mon rook-ceph-mon-a already exists. updating if needed 2022-03-10T03:11:05.571 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:11:05.581537 I | op-k8sutil: deployment "rook-ceph-mon-a" did not change, nothing to update 2022-03-10T03:11:05.575 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:11:05.586359 I | op-mon: deployment for mon rook-ceph-mon-b already exists. updating if needed 2022-03-10T03:11:05.584 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:11:05.594993 I | op-k8sutil: deployment "rook-ceph-mon-b" did not change, nothing to update 2022-03-10T03:11:05.785 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:11:05.795141 I | op-mon: updating maxMonID from 1 to 2 after committing mon "c" 2022-03-10T03:11:06.091 INFO:teuthology.orchestra.run.smithi037.stdout: 2022-03-10T03:11:06.092 INFO:teuthology.orchestra.run.smithi037.stderr:no valid command found; 10 closest matches: 2022-03-10T03:11:06.092 INFO:teuthology.orchestra.run.smithi037.stderr:pg stat 2022-03-10T03:11:06.092 INFO:teuthology.orchestra.run.smithi037.stderr:pg getmap 2022-03-10T03:11:06.092 INFO:teuthology.orchestra.run.smithi037.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-10T03:11:06.093 INFO:teuthology.orchestra.run.smithi037.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-10T03:11:06.093 INFO:teuthology.orchestra.run.smithi037.stderr:pg dump_pools_json 2022-03-10T03:11:06.093 INFO:teuthology.orchestra.run.smithi037.stderr:pg ls-by-pool [...] 2022-03-10T03:11:06.093 INFO:teuthology.orchestra.run.smithi037.stderr:pg ls-by-primary [] [...] 2022-03-10T03:11:06.094 INFO:teuthology.orchestra.run.smithi037.stderr:pg ls-by-osd [] [...] 2022-03-10T03:11:06.094 INFO:teuthology.orchestra.run.smithi037.stderr:pg ls [] [...] 2022-03-10T03:11:06.094 INFO:teuthology.orchestra.run.smithi037.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-10T03:11:06.094 INFO:teuthology.orchestra.run.smithi037.stderr:Error EINVAL: invalid command 2022-03-10T03:11:06.096 INFO:teuthology.orchestra.run.smithi037.stderr:command terminated with exit code 22 2022-03-10T03:11:06.099 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-10T03:11:06.520 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:11:06.530302 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.100.186.100:6789","10.111.132.151:6789","10.101.46.100:6789"]}] data:a=10.100.186.100:6789,b=10.111.132.151:6789,c=10.101.46.100:6789 mapping:{"node":{"a":{"Name":"smithi038","Hostname":"smithi038","Address":"172.21.15.38"},"b":{"Name":"smithi195","Hostname":"smithi195","Address":"172.21.15.195"},"c":{"Name":"smithi037","Hostname":"smithi037","Address":"172.21.15.37"}}} maxMonId:2] 2022-03-10T03:11:06.520 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:11:06.530337 I | op-mon: waiting for mon quorum with [a b c] 2022-03-10T03:11:07.124 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:11:07.134653 I | op-mon: mon c is not yet running 2022-03-10T03:11:07.125 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:11:07.134692 I | op-mon: mons running: [a b] 2022-03-10T03:11:07.359 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:11:07.369173 I | op-mon: Monitors in quorum: [a b] 2022-03-10T03:11:07.593 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:11:07.369204 I | op-mon: mons created: 3 2022-03-10T03:11:07.612 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:11:07.622333 I | op-mon: waiting for mon quorum with [a b c] 2022-03-10T03:11:07.739 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:11:07.749352 I | op-mon: mon c is not yet running 2022-03-10T03:11:07.739 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:11:07.749389 I | op-mon: mons running: [a b] 2022-03-10T03:11:12.761 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:11:12.771633 I | op-mon: mon c is not yet running 2022-03-10T03:11:12.762 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:11:12.771672 I | op-mon: mons running: [a b] 2022-03-10T03:11:16.101 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph orch status -f json 2022-03-10T03:11:16.608 INFO:teuthology.orchestra.run.smithi037.stdout: 2022-03-10T03:11:16.609 INFO:teuthology.orchestra.run.smithi037.stderr:no valid command found; 10 closest matches: 2022-03-10T03:11:16.609 INFO:teuthology.orchestra.run.smithi037.stderr:pg stat 2022-03-10T03:11:16.609 INFO:teuthology.orchestra.run.smithi037.stderr:pg getmap 2022-03-10T03:11:16.609 INFO:teuthology.orchestra.run.smithi037.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-10T03:11:16.610 INFO:teuthology.orchestra.run.smithi037.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-10T03:11:16.610 INFO:teuthology.orchestra.run.smithi037.stderr:pg dump_pools_json 2022-03-10T03:11:16.610 INFO:teuthology.orchestra.run.smithi037.stderr:pg ls-by-pool [...] 2022-03-10T03:11:16.610 INFO:teuthology.orchestra.run.smithi037.stderr:pg ls-by-primary [] [...] 2022-03-10T03:11:16.611 INFO:teuthology.orchestra.run.smithi037.stderr:pg ls-by-osd [] [...] 2022-03-10T03:11:16.611 INFO:teuthology.orchestra.run.smithi037.stderr:pg ls [] [...] 2022-03-10T03:11:16.611 INFO:teuthology.orchestra.run.smithi037.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-10T03:11:16.611 INFO:teuthology.orchestra.run.smithi037.stderr:Error EINVAL: invalid command 2022-03-10T03:11:16.612 INFO:teuthology.orchestra.run.smithi037.stderr:command terminated with exit code 22 2022-03-10T03:11:16.615 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-10T03:11:17.781 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:11:17.790742 I | op-mon: mon c is not yet running 2022-03-10T03:11:17.781 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:11:17.790776 I | op-mon: mons running: [a b] 2022-03-10T03:11:22.802 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:11:22.812444 I | op-mon: mon c is not yet running 2022-03-10T03:11:22.802 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:11:22.812480 I | op-mon: mons running: [a b] 2022-03-10T03:11:26.616 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph orch status -f json 2022-03-10T03:11:27.114 INFO:teuthology.orchestra.run.smithi037.stdout: 2022-03-10T03:11:27.115 INFO:teuthology.orchestra.run.smithi037.stderr:no valid command found; 10 closest matches: 2022-03-10T03:11:27.115 INFO:teuthology.orchestra.run.smithi037.stderr:pg stat 2022-03-10T03:11:27.115 INFO:teuthology.orchestra.run.smithi037.stderr:pg getmap 2022-03-10T03:11:27.115 INFO:teuthology.orchestra.run.smithi037.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-10T03:11:27.116 INFO:teuthology.orchestra.run.smithi037.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-10T03:11:27.116 INFO:teuthology.orchestra.run.smithi037.stderr:pg dump_pools_json 2022-03-10T03:11:27.116 INFO:teuthology.orchestra.run.smithi037.stderr:pg ls-by-pool [...] 2022-03-10T03:11:27.116 INFO:teuthology.orchestra.run.smithi037.stderr:pg ls-by-primary [] [...] 2022-03-10T03:11:27.117 INFO:teuthology.orchestra.run.smithi037.stderr:pg ls-by-osd [] [...] 2022-03-10T03:11:27.117 INFO:teuthology.orchestra.run.smithi037.stderr:pg ls [] [...] 2022-03-10T03:11:27.117 INFO:teuthology.orchestra.run.smithi037.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-10T03:11:27.117 INFO:teuthology.orchestra.run.smithi037.stderr:Error EINVAL: invalid command 2022-03-10T03:11:27.119 INFO:teuthology.orchestra.run.smithi037.stderr:command terminated with exit code 22 2022-03-10T03:11:27.120 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-10T03:11:28.531 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:11:28.542068 I | op-mon: mon c is not yet running 2022-03-10T03:11:28.532 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:11:28.542107 I | op-mon: mons running: [a b] 2022-03-10T03:11:34.375 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:11:34.385984 I | op-mon: mon c is not yet running 2022-03-10T03:11:34.375 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:11:34.386022 I | op-mon: mons running: [a b] 2022-03-10T03:11:37.122 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph orch status -f json 2022-03-10T03:11:37.593 INFO:teuthology.orchestra.run.smithi037.stdout: 2022-03-10T03:11:37.594 INFO:teuthology.orchestra.run.smithi037.stderr:no valid command found; 10 closest matches: 2022-03-10T03:11:37.594 INFO:teuthology.orchestra.run.smithi037.stderr:pg stat 2022-03-10T03:11:37.594 INFO:teuthology.orchestra.run.smithi037.stderr:pg getmap 2022-03-10T03:11:37.595 INFO:teuthology.orchestra.run.smithi037.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-10T03:11:37.595 INFO:teuthology.orchestra.run.smithi037.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-10T03:11:37.595 INFO:teuthology.orchestra.run.smithi037.stderr:pg dump_pools_json 2022-03-10T03:11:37.595 INFO:teuthology.orchestra.run.smithi037.stderr:pg ls-by-pool [...] 2022-03-10T03:11:37.596 INFO:teuthology.orchestra.run.smithi037.stderr:pg ls-by-primary [] [...] 2022-03-10T03:11:37.596 INFO:teuthology.orchestra.run.smithi037.stderr:pg ls-by-osd [] [...] 2022-03-10T03:11:37.596 INFO:teuthology.orchestra.run.smithi037.stderr:pg ls [] [...] 2022-03-10T03:11:37.596 INFO:teuthology.orchestra.run.smithi037.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-10T03:11:37.596 INFO:teuthology.orchestra.run.smithi037.stderr:Error EINVAL: invalid command 2022-03-10T03:11:37.598 INFO:teuthology.orchestra.run.smithi037.stderr:command terminated with exit code 22 2022-03-10T03:11:37.601 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-10T03:11:39.412 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:11:39.423268 I | op-mon: mon c is not yet running 2022-03-10T03:11:39.413 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:11:39.423302 I | op-mon: mons running: [a b] 2022-03-10T03:11:44.482 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:11:44.492611 I | op-mon: mons running: [a b c] 2022-03-10T03:11:47.603 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph orch status -f json 2022-03-10T03:11:49.145 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:11:49.156315 I | op-mon: Monitors in quorum: [a b c] 2022-03-10T03:11:49.146 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:11:49.156357 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-10T03:11:49.215 INFO:teuthology.orchestra.run.smithi037.stdout: 2022-03-10T03:11:49.215 INFO:teuthology.orchestra.run.smithi037.stderr:no valid command found; 10 closest matches: 2022-03-10T03:11:49.216 INFO:teuthology.orchestra.run.smithi037.stderr:pg stat 2022-03-10T03:11:49.216 INFO:teuthology.orchestra.run.smithi037.stderr:pg getmap 2022-03-10T03:11:49.216 INFO:teuthology.orchestra.run.smithi037.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-10T03:11:49.217 INFO:teuthology.orchestra.run.smithi037.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-10T03:11:49.217 INFO:teuthology.orchestra.run.smithi037.stderr:pg dump_pools_json 2022-03-10T03:11:49.217 INFO:teuthology.orchestra.run.smithi037.stderr:pg ls-by-pool [...] 2022-03-10T03:11:49.217 INFO:teuthology.orchestra.run.smithi037.stderr:pg ls-by-primary [] [...] 2022-03-10T03:11:49.217 INFO:teuthology.orchestra.run.smithi037.stderr:pg ls-by-osd [] [...] 2022-03-10T03:11:49.218 INFO:teuthology.orchestra.run.smithi037.stderr:pg ls [] [...] 2022-03-10T03:11:49.218 INFO:teuthology.orchestra.run.smithi037.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-10T03:11:49.218 INFO:teuthology.orchestra.run.smithi037.stderr:Error EINVAL: invalid command 2022-03-10T03:11:49.219 INFO:teuthology.orchestra.run.smithi037.stderr:command terminated with exit code 22 2022-03-10T03:11:49.220 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-10T03:11:49.367 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:11:49.378029 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-10T03:11:49.368 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:11:49.378056 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-03-10T03:11:49.594 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:11:49.604486 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-03-10T03:11:49.594 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:11:49.604518 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-10T03:11:49.808 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:11:49.818512 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-10T03:11:49.808 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:11:49.818541 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-03-10T03:11:50.028 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:11:50.038898 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-03-10T03:11:50.028 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:11:50.038964 I | op-config: deleting "log file" option from the mon configuration database 2022-03-10T03:11:50.249 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:11:50.260425 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-03-10T03:11:50.250 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:11:50.260459 I | cephclient: getting or creating ceph auth key "client.csi-rbd-provisioner" 2022-03-10T03:11:50.570 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:11:50.581175 I | cephclient: getting or creating ceph auth key "client.csi-rbd-node" 2022-03-10T03:11:50.902 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:11:50.912795 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-provisioner" 2022-03-10T03:11:51.225 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:11:51.236101 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-node" 2022-03-10T03:11:51.642 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:11:51.653504 I | ceph-csi: created kubernetes csi secrets for cluster "rook-ceph" 2022-03-10T03:11:51.643 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:11:51.653542 I | cephclient: getting or creating ceph auth key "client.crash" 2022-03-10T03:11:51.972 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:11:51.983329 I | ceph-crashcollector-controller: created kubernetes crash collector secret for cluster "rook-ceph" 2022-03-10T03:11:52.192 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:11:52.203037 I | cephclient: successfully enabled msgr2 protocol 2022-03-10T03:11:52.193 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:11:52.203072 I | cephclient: create rbd-mirror bootstrap peer token "client.rbd-mirror-peer" 2022-03-10T03:11:52.193 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:11:52.203080 I | cephclient: getting or creating ceph auth key "client.rbd-mirror-peer" 2022-03-10T03:11:52.563 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:11:52.574141 I | cephclient: successfully created rbd-mirror bootstrap peer token for cluster "rook-ceph" 2022-03-10T03:11:52.580 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:11:52.591036 I | op-mgr: start running mgr 2022-03-10T03:11:52.580 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:11:52.591068 I | cephclient: getting or creating ceph auth key "mgr.a" 2022-03-10T03:11:53.545 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:11:53.556698 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi037": the object has been modified; please apply your changes to the latest version and try again 2022-03-10T03:11:59.222 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph orch status -f json 2022-03-10T03:11:59.724 INFO:teuthology.orchestra.run.smithi037.stdout: 2022-03-10T03:11:59.725 INFO:teuthology.orchestra.run.smithi037.stderr:no valid command found; 10 closest matches: 2022-03-10T03:11:59.725 INFO:teuthology.orchestra.run.smithi037.stderr:pg stat 2022-03-10T03:11:59.726 INFO:teuthology.orchestra.run.smithi037.stderr:pg getmap 2022-03-10T03:11:59.726 INFO:teuthology.orchestra.run.smithi037.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-10T03:11:59.726 INFO:teuthology.orchestra.run.smithi037.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-10T03:11:59.726 INFO:teuthology.orchestra.run.smithi037.stderr:pg dump_pools_json 2022-03-10T03:11:59.726 INFO:teuthology.orchestra.run.smithi037.stderr:pg ls-by-pool [...] 2022-03-10T03:11:59.727 INFO:teuthology.orchestra.run.smithi037.stderr:pg ls-by-primary [] [...] 2022-03-10T03:11:59.727 INFO:teuthology.orchestra.run.smithi037.stderr:pg ls-by-osd [] [...] 2022-03-10T03:11:59.727 INFO:teuthology.orchestra.run.smithi037.stderr:pg ls [] [...] 2022-03-10T03:11:59.727 INFO:teuthology.orchestra.run.smithi037.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-10T03:11:59.728 INFO:teuthology.orchestra.run.smithi037.stderr:Error EINVAL: invalid command 2022-03-10T03:11:59.729 INFO:teuthology.orchestra.run.smithi037.stderr:command terminated with exit code 22 2022-03-10T03:11:59.731 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-10T03:12:03.761 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:03.772721 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi038": the object has been modified; please apply your changes to the latest version and try again 2022-03-10T03:12:09.732 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph orch status -f json 2022-03-10T03:12:10.302 INFO:teuthology.orchestra.run.smithi037.stdout: 2022-03-10T03:12:10.304 INFO:teuthology.orchestra.run.smithi037.stderr:Error ENOENT: No orchestrator configured (try `ceph orch set backend`) 2022-03-10T03:12:10.309 INFO:teuthology.orchestra.run.smithi037.stderr:command terminated with exit code 2 2022-03-10T03:12:10.311 DEBUG:teuthology.orchestra.run:got remote process result: 2 2022-03-10T03:12:14.421 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:14.432344 I | op-k8sutil: finished waiting for updated deployment "rook-ceph-mgr-a" 2022-03-10T03:12:14.426 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:14.437186 I | op-mgr: setting services to point to mgr "a" 2022-03-10T03:12:14.514 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:14.525769 I | op-mgr: no need to update service "rook-ceph-mgr" 2022-03-10T03:12:14.515 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:14.525903 I | op-mgr: successful modules: balancer 2022-03-10T03:12:14.526 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:14.535341 I | op-osd: start running osds in namespace "rook-ceph" 2022-03-10T03:12:14.526 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:14.535388 W | op-osd: useAllNodes is set to false and no nodes, storageClassDevicesets or volumeSources are specified, no OSD pods are going to be created 2022-03-10T03:12:14.526 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:14.535400 I | op-osd: wait timeout for healthy OSDs during upgrade or restart is "10m0s" 2022-03-10T03:12:14.548 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:14.559860 I | op-osd: start provisioning the OSDs on PVCs, if needed 2022-03-10T03:12:14.569 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:14.563357 I | op-osd: no storageClassDeviceSets defined to configure OSDs on PVCs 2022-03-10T03:12:14.569 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:14.563389 I | op-osd: start provisioning the OSDs on nodes, if needed 2022-03-10T03:12:14.570 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:14.563395 I | op-osd: no nodes are defined for configuring OSDs on raw devices 2022-03-10T03:12:15.002 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:15.013000 I | op-osd: finished running OSDs in namespace "rook-ceph" 2022-03-10T03:12:15.002 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:15.013036 I | ceph-cluster-controller: done reconciling ceph cluster in namespace "rook-ceph" 2022-03-10T03:12:15.077 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:15.087927 I | ceph-cluster-controller: enabling ceph mon monitoring goroutine for cluster "rook-ceph" 2022-03-10T03:12:15.077 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:15.087994 I | ceph-cluster-controller: enabling ceph osd monitoring goroutine for cluster "rook-ceph" 2022-03-10T03:12:15.077 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:15.088018 I | ceph-cluster-controller: enabling ceph status monitoring goroutine for cluster "rook-ceph" 2022-03-10T03:12:15.294 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:15.305277 I | op-mgr: successful modules: mgr module(s) from the spec 2022-03-10T03:12:15.300 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:15.311797 I | op-mgr: successful modules: dashboard 2022-03-10T03:12:15.300 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:15.311828 I | op-mgr: successful modules: prometheus 2022-03-10T03:12:19.962 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:19.973904 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi195": the object has been modified; please apply your changes to the latest version and try again 2022-03-10T03:12:20.313 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph orch status -f json 2022-03-10T03:12:24.464 INFO:teuthology.orchestra.run.smithi037.stdout: 2022-03-10T03:12:24.467 INFO:teuthology.orchestra.run.smithi037.stderr:Error ENOENT: No orchestrator configured (try `ceph orch set backend`) 2022-03-10T03:12:24.471 INFO:teuthology.orchestra.run.smithi037.stderr:command terminated with exit code 2 2022-03-10T03:12:24.473 DEBUG:teuthology.orchestra.run:got remote process result: 2 2022-03-10T03:12:24.630 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:24.641748 I | op-mgr: successful modules: orchestrator modules 2022-03-10T03:12:34.475 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph orch status -f json 2022-03-10T03:12:35.147 INFO:teuthology.orchestra.run.smithi037.stdout: 2022-03-10T03:12:35.147 INFO:teuthology.orchestra.run.smithi037.stdout:{"available": true, "backend": "rook"} 2022-03-10T03:12:35.178 INFO:tasks.rook: mgr/rook orchestrator is active 2022-03-10T03:12:35.179 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph config set mgr mgr/rook/storage_class scratch 2022-03-10T03:12:35.876 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph orch apply osd --all-available-devices 2022-03-10T03:12:36.623 INFO:teuthology.orchestra.run.smithi037.stdout:['Success'] 2022-03-10T03:12:36.633 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:36.635335 I | ceph-cluster-controller: CR has changed for "rook-ceph". diff= v1.ClusterSpec{ 2022-03-10T03:12:36.633 INFO:tasks.rook.operator.smithi037.stdout: CephVersion: {Image: "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", AllowUnsupported: true}, 2022-03-10T03:12:36.633 INFO:tasks.rook.operator.smithi037.stdout: Storage: v1.StorageScopeSpec{ 2022-03-10T03:12:36.634 INFO:tasks.rook.operator.smithi037.stdout: ... // 3 identical fields 2022-03-10T03:12:36.634 INFO:tasks.rook.operator.smithi037.stdout: Config: nil, 2022-03-10T03:12:36.634 INFO:tasks.rook.operator.smithi037.stdout: Selection: {}, 2022-03-10T03:12:36.635 INFO:tasks.rook.operator.smithi037.stdout:- StorageClassDeviceSets: nil, 2022-03-10T03:12:36.635 INFO:tasks.rook.operator.smithi037.stdout:+ StorageClassDeviceSets: []v1.StorageClassDeviceSet{ 2022-03-10T03:12:36.635 INFO:tasks.rook.operator.smithi037.stdout:+ { 2022-03-10T03:12:36.636 INFO:tasks.rook.operator.smithi037.stdout:+ Name: "smithi037-lv-1", 2022-03-10T03:12:36.636 INFO:tasks.rook.operator.smithi037.stdout:+ Count: 1, 2022-03-10T03:12:36.636 INFO:tasks.rook.operator.smithi037.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-10T03:12:36.636 INFO:tasks.rook.operator.smithi037.stdout:+ }, 2022-03-10T03:12:36.637 INFO:tasks.rook.operator.smithi037.stdout:+ { 2022-03-10T03:12:36.637 INFO:tasks.rook.operator.smithi037.stdout:+ Name: "smithi037-lv-2", 2022-03-10T03:12:36.637 INFO:tasks.rook.operator.smithi037.stdout:+ Count: 1, 2022-03-10T03:12:36.637 INFO:tasks.rook.operator.smithi037.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-10T03:12:36.638 INFO:tasks.rook.operator.smithi037.stdout:+ }, 2022-03-10T03:12:36.638 INFO:tasks.rook.operator.smithi037.stdout:+ { 2022-03-10T03:12:36.638 INFO:tasks.rook.operator.smithi037.stdout:+ Name: "smithi037-lv-3", 2022-03-10T03:12:36.638 INFO:tasks.rook.operator.smithi037.stdout:+ Count: 1, 2022-03-10T03:12:36.639 INFO:tasks.rook.operator.smithi037.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-10T03:12:36.639 INFO:tasks.rook.operator.smithi037.stdout:+ }, 2022-03-10T03:12:36.639 INFO:tasks.rook.operator.smithi037.stdout:+ { 2022-03-10T03:12:36.639 INFO:tasks.rook.operator.smithi037.stdout:+ Name: "smithi037-lv-4", 2022-03-10T03:12:36.639 INFO:tasks.rook.operator.smithi037.stdout:+ Count: 1, 2022-03-10T03:12:36.640 INFO:tasks.rook.operator.smithi037.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-10T03:12:36.640 INFO:tasks.rook.operator.smithi037.stdout:+ }, 2022-03-10T03:12:36.640 INFO:tasks.rook.operator.smithi037.stdout:+ { 2022-03-10T03:12:36.640 INFO:tasks.rook.operator.smithi037.stdout:+ Name: "smithi038-lv-1", 2022-03-10T03:12:36.641 INFO:tasks.rook.operator.smithi037.stdout:+ Count: 1, 2022-03-10T03:12:36.641 INFO:tasks.rook.operator.smithi037.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-10T03:12:36.641 INFO:tasks.rook.operator.smithi037.stdout:+ }, 2022-03-10T03:12:36.641 INFO:tasks.rook.operator.smithi037.stdout:+ { 2022-03-10T03:12:36.641 INFO:tasks.rook.operator.smithi037.stdout:+ Name: "smithi038-lv-2", 2022-03-10T03:12:36.642 INFO:tasks.rook.operator.smithi037.stdout:+ Count: 1, 2022-03-10T03:12:36.642 INFO:tasks.rook.operator.smithi037.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-10T03:12:36.642 INFO:tasks.rook.operator.smithi037.stdout:+ }, 2022-03-10T03:12:36.642 INFO:tasks.rook.operator.smithi037.stdout:+ { 2022-03-10T03:12:36.642 INFO:tasks.rook.operator.smithi037.stdout:+ Name: "smithi038-lv-3", 2022-03-10T03:12:36.643 INFO:tasks.rook.operator.smithi037.stdout:+ Count: 1, 2022-03-10T03:12:36.643 INFO:tasks.rook.operator.smithi037.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-10T03:12:36.643 INFO:tasks.rook.operator.smithi037.stdout:+ }, 2022-03-10T03:12:36.643 INFO:tasks.rook.operator.smithi037.stdout:+ { 2022-03-10T03:12:36.643 INFO:tasks.rook.operator.smithi037.stdout:+ Name: "smithi038-lv-4", 2022-03-10T03:12:36.644 INFO:tasks.rook.operator.smithi037.stdout:+ Count: 1, 2022-03-10T03:12:36.644 INFO:tasks.rook.operator.smithi037.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-10T03:12:36.644 INFO:tasks.rook.operator.smithi037.stdout:+ }, 2022-03-10T03:12:36.645 INFO:tasks.rook.operator.smithi037.stdout:+ { 2022-03-10T03:12:36.645 INFO:tasks.rook.operator.smithi037.stdout:+ Name: "smithi195-lv-1", 2022-03-10T03:12:36.645 INFO:tasks.rook.operator.smithi037.stdout:+ Count: 1, 2022-03-10T03:12:36.646 INFO:tasks.rook.operator.smithi037.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-10T03:12:36.646 INFO:tasks.rook.operator.smithi037.stdout:+ }, 2022-03-10T03:12:36.646 INFO:tasks.rook.operator.smithi037.stdout:+ { 2022-03-10T03:12:36.646 INFO:tasks.rook.operator.smithi037.stdout:+ Name: "smithi195-lv-2", 2022-03-10T03:12:36.647 INFO:tasks.rook.operator.smithi037.stdout:+ Count: 1, 2022-03-10T03:12:36.647 INFO:tasks.rook.operator.smithi037.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-10T03:12:36.647 INFO:tasks.rook.operator.smithi037.stdout:+ }, 2022-03-10T03:12:36.647 INFO:tasks.rook.operator.smithi037.stdout:+ { 2022-03-10T03:12:36.648 INFO:tasks.rook.operator.smithi037.stdout:+ Name: "smithi195-lv-3", 2022-03-10T03:12:36.648 INFO:tasks.rook.operator.smithi037.stdout:+ Count: 1, 2022-03-10T03:12:36.648 INFO:tasks.rook.operator.smithi037.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-10T03:12:36.648 INFO:tasks.rook.operator.smithi037.stdout:+ }, 2022-03-10T03:12:36.648 INFO:tasks.rook.operator.smithi037.stdout:+ { 2022-03-10T03:12:36.649 INFO:tasks.rook.operator.smithi037.stdout:+ Name: "smithi195-lv-4", 2022-03-10T03:12:36.649 INFO:tasks.rook.operator.smithi037.stdout:+ Count: 1, 2022-03-10T03:12:36.649 INFO:tasks.rook.operator.smithi037.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-10T03:12:36.649 INFO:tasks.rook.operator.smithi037.stdout:+ }, 2022-03-10T03:12:36.650 INFO:tasks.rook.operator.smithi037.stdout:+ }, 2022-03-10T03:12:36.650 INFO:tasks.rook.operator.smithi037.stdout: }, 2022-03-10T03:12:36.650 INFO:tasks.rook.operator.smithi037.stdout: Annotations: nil, 2022-03-10T03:12:36.650 INFO:tasks.rook.operator.smithi037.stdout: Labels: nil, 2022-03-10T03:12:36.651 INFO:tasks.rook.operator.smithi037.stdout: ... // 20 identical fields 2022-03-10T03:12:36.651 INFO:tasks.rook.operator.smithi037.stdout: } 2022-03-10T03:12:36.651 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:36.635474 I | operator: reloading operator's CRDs manager, cancelling all orchestrations! 2022-03-10T03:12:36.651 INFO:tasks.rook.operator.smithi037.stdout:I0310 03:12:36.635920 1 manager.go:148] objectbucket.io/provisioner-manager "msg"="stopping provisioner" "name"="rook-ceph.ceph.rook.io/bucket" "reason"={} 2022-03-10T03:12:36.652 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:36.636137 I | op-mon: stopping monitoring of mons in namespace "rook-ceph" 2022-03-10T03:12:36.652 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:36.636177 I | op-osd: stopping monitoring of OSDs in namespace "rook-ceph" 2022-03-10T03:12:36.652 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:36.636195 I | ceph-cluster-controller: stopping monitoring of ceph status 2022-03-10T03:12:36.652 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:36.636291 I | operator: successfully started the controller-runtime manager 2022-03-10T03:12:36.653 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:36.658501 I | op-k8sutil: ROOK_CURRENT_NAMESPACE_ONLY="false" (env var) 2022-03-10T03:12:36.653 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:36.658521 I | operator: watching all namespaces for Ceph CRs 2022-03-10T03:12:36.653 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:36.658544 I | operator: setting up schemes 2022-03-10T03:12:36.655 INFO:tasks.rook:Waiting for 12 OSDs 2022-03-10T03:12:36.655 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph osd stat -f json 2022-03-10T03:12:36.656 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:36.660728 I | operator: setting up the controller-runtime manager 2022-03-10T03:12:36.955 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:36.965287 I | operator: looking for admission webhook secret "rook-ceph-admission-controller" 2022-03-10T03:12:36.960 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:36.971121 I | operator: admission webhook secret "rook-ceph-admission-controller" not found. proceeding without the admission controller 2022-03-10T03:12:36.960 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:36.971163 I | ceph-cluster-controller: successfully started 2022-03-10T03:12:36.960 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:36.971229 I | ceph-cluster-controller: enabling hotplug orchestration 2022-03-10T03:12:36.961 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:36.971256 I | ceph-crashcollector-controller: successfully started 2022-03-10T03:12:36.961 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:36.971291 I | ceph-block-pool-controller: successfully started 2022-03-10T03:12:36.961 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:36.971330 I | ceph-object-store-user-controller: successfully started 2022-03-10T03:12:36.962 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:36.971362 I | ceph-object-realm-controller: successfully started 2022-03-10T03:12:36.962 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:36.971393 I | ceph-object-zonegroup-controller: successfully started 2022-03-10T03:12:36.963 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:36.971418 I | ceph-object-zone-controller: successfully started 2022-03-10T03:12:36.964 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:36.971568 I | ceph-object-controller: successfully started 2022-03-10T03:12:36.965 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:36.971617 I | ceph-file-controller: successfully started 2022-03-10T03:12:36.965 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:36.971666 I | ceph-nfs-controller: successfully started 2022-03-10T03:12:36.965 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:36.971711 I | ceph-rbd-mirror-controller: successfully started 2022-03-10T03:12:36.965 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:36.971755 I | ceph-client-controller: successfully started 2022-03-10T03:12:36.965 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:36.971793 I | ceph-filesystem-mirror-controller: successfully started 2022-03-10T03:12:36.966 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:36.971830 I | operator: rook-ceph-operator-config-controller successfully started 2022-03-10T03:12:36.966 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:36.971860 I | ceph-csi: rook-ceph-operator-csi-controller successfully started 2022-03-10T03:12:36.967 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:36.971888 I | op-bucket-prov: rook-ceph-operator-bucket-controller successfully started 2022-03-10T03:12:36.967 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:36.971916 I | ceph-bucket-topic: successfully started 2022-03-10T03:12:36.967 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:36.971940 I | ceph-bucket-notification: successfully started 2022-03-10T03:12:36.968 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:36.971962 I | ceph-bucket-notification: successfully started 2022-03-10T03:12:36.968 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:36.971975 I | ceph-fs-subvolumegroup-controller: successfully started 2022-03-10T03:12:36.968 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:36.972867 I | operator: starting the controller-runtime manager 2022-03-10T03:12:37.066 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:37.076892 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-03-10T03:12:37.066 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:37.076996 I | op-k8sutil: ROOK_CEPH_COMMANDS_TIMEOUT_SECONDS="15" (configmap) 2022-03-10T03:12:37.066 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:37.077062 I | op-k8sutil: ROOK_LOG_LEVEL="INFO" (configmap) 2022-03-10T03:12:37.067 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:37.077083 I | op-k8sutil: ROOK_ENABLE_DISCOVERY_DAEMON="false" (configmap) 2022-03-10T03:12:37.070 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:37.081790 I | operator: rook-ceph-operator-config-controller done reconciling 2022-03-10T03:12:37.074 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:37.083324 I | op-mon: parsing mon endpoints: a=10.100.186.100:6789,b=10.111.132.151:6789,c=10.101.46.100:6789 2022-03-10T03:12:37.074 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:37.083375 I | op-k8sutil: ROOK_OBC_WATCH_OPERATOR_NAMESPACE="true" (configmap) 2022-03-10T03:12:37.074 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:37.083395 I | op-bucket-prov: ceph bucket provisioner launched watching for provisioner "rook-ceph.ceph.rook.io/bucket" 2022-03-10T03:12:37.075 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:37.084240 I | op-mon: parsing mon endpoints: a=10.100.186.100:6789,b=10.111.132.151:6789,c=10.101.46.100:6789 2022-03-10T03:12:37.075 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:37.084601 I | op-bucket-prov: successfully reconciled bucket provisioner 2022-03-10T03:12:37.075 INFO:tasks.rook.operator.smithi037.stdout:I0310 03:12:37.084668 1 manager.go:135] objectbucket.io/provisioner-manager "msg"="starting provisioner" "name"="rook-ceph.ceph.rook.io/bucket" 2022-03-10T03:12:37.086 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:37.098127 I | ceph-csi: successfully created csi config map "rook-ceph-csi-config" 2022-03-10T03:12:37.089 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:37.100243 I | ceph-spec: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3... 2022-03-10T03:12:37.119 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:37.128466 I | op-k8sutil: ROOK_CSI_ENABLE_RBD="true" (configmap) 2022-03-10T03:12:37.119 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:37.128497 I | op-k8sutil: ROOK_CSI_ENABLE_CEPHFS="true" (configmap) 2022-03-10T03:12:37.119 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:37.128505 I | op-k8sutil: ROOK_CSI_ALLOW_UNSUPPORTED_VERSION="false" (configmap) 2022-03-10T03:12:37.120 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:37.128512 I | op-k8sutil: ROOK_CSI_ENABLE_GRPC_METRICS="false" (configmap) 2022-03-10T03:12:37.120 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:37.128520 I | op-k8sutil: CSI_ENABLE_HOST_NETWORK="true" (default) 2022-03-10T03:12:37.120 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:37.128528 I | op-k8sutil: ROOK_CSI_CEPH_IMAGE="quay.io/cephcsi/cephcsi:v3.5.1" (default) 2022-03-10T03:12:37.120 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:37.128542 I | op-k8sutil: ROOK_CSI_REGISTRAR_IMAGE="k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.5.0" (default) 2022-03-10T03:12:37.120 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:37.128555 I | op-k8sutil: ROOK_CSI_PROVISIONER_IMAGE="k8s.gcr.io/sig-storage/csi-provisioner:v3.1.0" (default) 2022-03-10T03:12:37.121 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:37.128568 I | op-k8sutil: ROOK_CSI_ATTACHER_IMAGE="k8s.gcr.io/sig-storage/csi-attacher:v3.4.0" (default) 2022-03-10T03:12:37.121 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:37.128576 I | op-k8sutil: ROOK_CSI_SNAPSHOTTER_IMAGE="k8s.gcr.io/sig-storage/csi-snapshotter:v5.0.1" (default) 2022-03-10T03:12:37.121 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:37.128583 I | op-k8sutil: ROOK_CSI_KUBELET_DIR_PATH="/var/lib/kubelet" (default) 2022-03-10T03:12:37.121 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:37.128598 I | op-k8sutil: CSI_VOLUME_REPLICATION_IMAGE="quay.io/csiaddons/volumereplication-operator:v0.3.0" (default) 2022-03-10T03:12:37.122 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:37.128611 I | op-k8sutil: ROOK_CSIADDONS_IMAGE="quay.io/csiaddons/k8s-sidecar:v0.2.1" (default) 2022-03-10T03:12:37.122 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:37.128621 I | op-k8sutil: ROOK_CSI_CEPHFS_POD_LABELS="" (default) 2022-03-10T03:12:37.122 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:37.128632 I | op-k8sutil: ROOK_CSI_RBD_POD_LABELS="" (default) 2022-03-10T03:12:37.122 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:37.128647 I | ceph-csi: detecting the ceph csi image version for image "quay.io/cephcsi/cephcsi:v3.5.1" 2022-03-10T03:12:37.123 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:37.128818 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-03-10T03:12:37.123 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:37.128834 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-10T03:12:37.262 INFO:teuthology.orchestra.run.smithi037.stdout: 2022-03-10T03:12:37.282 INFO:teuthology.orchestra.run.smithi037.stdout:{"epoch":2,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2022-03-10T03:12:37.282 INFO:tasks.rook: have 0/12 OSDs 2022-03-10T03:12:42.822 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:42.830432 I | ceph-csi: Detected ceph CSI image version: "v3.5.1" 2022-03-10T03:12:42.822 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:42.830473 I | op-k8sutil: CSI_FORCE_CEPHFS_KERNEL_CLIENT="true" (configmap) 2022-03-10T03:12:42.822 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:42.830479 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default) 2022-03-10T03:12:42.822 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:42.830481 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default) 2022-03-10T03:12:42.823 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:42.830491 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default) 2022-03-10T03:12:42.823 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:42.830493 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default) 2022-03-10T03:12:42.823 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:42.830497 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default) 2022-03-10T03:12:42.823 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:42.830500 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default) 2022-03-10T03:12:42.824 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:42.830505 I | op-k8sutil: CSIADDONS_PORT="9070" (default) 2022-03-10T03:12:42.824 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:42.830509 I | op-k8sutil: CSIADDONS_PORT="9070" (default) 2022-03-10T03:12:42.824 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:42.830513 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default) 2022-03-10T03:12:42.824 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:42.830517 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default) 2022-03-10T03:12:42.824 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:42.830521 I | op-k8sutil: CSI_PLUGIN_PRIORITY_CLASSNAME="" (default) 2022-03-10T03:12:42.825 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:42.830524 I | op-k8sutil: CSI_PROVISIONER_PRIORITY_CLASSNAME="" (default) 2022-03-10T03:12:42.825 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:42.830527 I | op-k8sutil: CSI_ENABLE_OMAP_GENERATOR="false" (default) 2022-03-10T03:12:42.825 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:42.830536 I | op-k8sutil: CSI_ENABLE_RBD_SNAPSHOTTER="true" (configmap) 2022-03-10T03:12:42.825 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:42.830539 I | op-k8sutil: CSI_ENABLE_CEPHFS_SNAPSHOTTER="true" (configmap) 2022-03-10T03:12:42.826 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:42.830542 I | op-k8sutil: CSI_ENABLE_VOLUME_REPLICATION="false" (configmap) 2022-03-10T03:12:42.826 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:42.830545 I | op-k8sutil: CSI_ENABLE_CSIADDONS="false" (configmap) 2022-03-10T03:12:42.826 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:42.830548 I | op-k8sutil: CSI_CEPHFS_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-03-10T03:12:42.827 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:42.830551 I | op-k8sutil: CSI_RBD_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-03-10T03:12:42.827 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:42.830554 I | op-k8sutil: CSI_PLUGIN_ENABLE_SELINUX_HOST_MOUNT="false" (configmap) 2022-03-10T03:12:42.827 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:42.830557 I | ceph-csi: Kubernetes version is 1.23 2022-03-10T03:12:42.827 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:42.830561 I | op-k8sutil: ROOK_CSI_RESIZER_IMAGE="k8s.gcr.io/sig-storage/csi-resizer:v1.4.0" (default) 2022-03-10T03:12:42.827 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:42.830564 I | op-k8sutil: CSI_LOG_LEVEL="" (default) 2022-03-10T03:12:42.837 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:42.842435 I | op-k8sutil: CSI_PROVISIONER_REPLICAS="2" (configmap) 2022-03-10T03:12:42.841 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:42.851096 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-03-10T03:12:42.841 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:42.851116 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-10T03:12:42.841 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:42.851120 I | op-k8sutil: CSI_PLUGIN_TOLERATIONS="" (default) 2022-03-10T03:12:42.841 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:42.851123 I | op-k8sutil: CSI_PLUGIN_NODE_AFFINITY="" (default) 2022-03-10T03:12:42.842 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:42.851125 I | op-k8sutil: CSI_RBD_PLUGIN_TOLERATIONS="" (default) 2022-03-10T03:12:42.842 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:42.851128 I | op-k8sutil: CSI_RBD_PLUGIN_NODE_AFFINITY="" (default) 2022-03-10T03:12:42.842 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:42.851131 I | op-k8sutil: CSI_RBD_PLUGIN_RESOURCE="" (default) 2022-03-10T03:12:42.941 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:42.951436 I | op-k8sutil: CSI_RBD_PROVISIONER_TOLERATIONS="" (default) 2022-03-10T03:12:42.941 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:42.951465 I | op-k8sutil: CSI_RBD_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-10T03:12:42.942 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:42.951476 I | op-k8sutil: CSI_RBD_PROVISIONER_RESOURCE="" (default) 2022-03-10T03:12:43.061 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:43.071271 I | ceph-csi: successfully started CSI Ceph RBD driver 2022-03-10T03:12:43.641 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:43.651090 I | ceph-spec: detected ceph image version: "17.0.0-11025 quincy" 2022-03-10T03:12:43.641 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:43.651119 I | ceph-cluster-controller: validating ceph version from provided image 2022-03-10T03:12:43.642 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:43.651130 W | ceph-cluster-controller: unsupported ceph version detected: "17.0.0-11025 quincy", pursuing 2022-03-10T03:12:43.834 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:43.843858 I | op-k8sutil: CSI_CEPHFS_PLUGIN_TOLERATIONS="" (default) 2022-03-10T03:12:43.835 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:43.843891 I | op-k8sutil: CSI_CEPHFS_PLUGIN_NODE_AFFINITY="" (default) 2022-03-10T03:12:43.835 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:43.843902 I | op-k8sutil: CSI_CEPHFS_PLUGIN_RESOURCE="" (default) 2022-03-10T03:12:43.840 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:43.847288 I | op-mon: parsing mon endpoints: a=10.100.186.100:6789,b=10.111.132.151:6789,c=10.101.46.100:6789 2022-03-10T03:12:43.882 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:43.891721 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-10T03:12:43.888 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:43.892199 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-10T03:12:43.948 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:43.957899 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_TOLERATIONS="" (default) 2022-03-10T03:12:43.948 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:43.957927 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-10T03:12:43.948 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:43.957937 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_RESOURCE="" (default) 2022-03-10T03:12:43.989 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:43.999595 I | ceph-csi: successfully started CSI CephFS driver 2022-03-10T03:12:44.182 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:44.192523 I | op-k8sutil: CSI_RBD_FSGROUPPOLICY="ReadWriteOnceWithFSType" (configmap) 2022-03-10T03:12:44.220 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:44.230523 I | ceph-csi: CSIDriver object updated for driver "rook-ceph.rbd.csi.ceph.com" 2022-03-10T03:12:44.220 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:44.230553 I | op-k8sutil: CSI_CEPHFS_FSGROUPPOLICY="ReadWriteOnceWithFSType" (configmap) 2022-03-10T03:12:44.254 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:44.264776 I | ceph-csi: CSIDriver object updated for driver "rook-ceph.cephfs.csi.ceph.com" 2022-03-10T03:12:44.304 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:44.314573 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-11025 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3" 2022-03-10T03:12:44.412 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:44.422251 I | op-mon: start running mons 2022-03-10T03:12:44.453 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:44.463277 I | op-mon: parsing mon endpoints: a=10.100.186.100:6789,b=10.111.132.151:6789,c=10.101.46.100:6789 2022-03-10T03:12:44.991 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:45.001397 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.100.186.100:6789","10.111.132.151:6789","10.101.46.100:6789"]}] data:a=10.100.186.100:6789,b=10.111.132.151:6789,c=10.101.46.100:6789 mapping:{"node":{"a":{"Name":"smithi038","Hostname":"smithi038","Address":"172.21.15.38"},"b":{"Name":"smithi195","Hostname":"smithi195","Address":"172.21.15.195"},"c":{"Name":"smithi037","Hostname":"smithi037","Address":"172.21.15.37"}}} maxMonId:2] 2022-03-10T03:12:45.591 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:45.601363 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-10T03:12:45.592 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:45.601595 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-10T03:12:47.192 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:47.201867 I | op-mon: targeting the mon count 3 2022-03-10T03:12:47.200 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:47.209710 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-10T03:12:47.284 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph osd stat -f json 2022-03-10T03:12:47.567 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:47.576359 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-10T03:12:47.567 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:47.576393 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-03-10T03:12:47.792 INFO:teuthology.orchestra.run.smithi037.stdout: 2022-03-10T03:12:47.812 INFO:teuthology.orchestra.run.smithi037.stdout:{"epoch":2,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2022-03-10T03:12:47.812 INFO:tasks.rook: have 0/12 OSDs 2022-03-10T03:12:47.915 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:47.924131 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-03-10T03:12:47.915 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:47.924162 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-10T03:12:48.273 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:48.282954 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-10T03:12:48.274 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:48.282990 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-03-10T03:12:48.707 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:48.716125 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-03-10T03:12:48.707 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:48.716160 I | op-config: deleting "log file" option from the mon configuration database 2022-03-10T03:12:49.071 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:49.080286 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-03-10T03:12:49.071 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:49.080310 I | op-mon: checking for basic quorum with existing mons 2022-03-10T03:12:49.208 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:49.217492 I | op-mon: mon "a" endpoint is [v2:10.100.186.100:3300,v1:10.100.186.100:6789] 2022-03-10T03:12:49.353 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:49.362780 I | op-mon: mon "b" endpoint is [v2:10.111.132.151:3300,v1:10.111.132.151:6789] 2022-03-10T03:12:49.593 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:49.602447 I | op-mon: mon "c" endpoint is [v2:10.101.46.100:3300,v1:10.101.46.100:6789] 2022-03-10T03:12:50.194 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:50.203431 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.101.46.100:6789","10.100.186.100:6789","10.111.132.151:6789"]}] data:a=10.100.186.100:6789,b=10.111.132.151:6789,c=10.101.46.100:6789 mapping:{"node":{"a":{"Name":"smithi038","Hostname":"smithi038","Address":"172.21.15.38"},"b":{"Name":"smithi195","Hostname":"smithi195","Address":"172.21.15.195"},"c":{"Name":"smithi037","Hostname":"smithi037","Address":"172.21.15.37"}}} maxMonId:2] 2022-03-10T03:12:50.793 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:50.802653 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-10T03:12:50.798 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:50.802994 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-10T03:12:51.214 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:51.223035 I | op-mon: deployment for mon rook-ceph-mon-a already exists. updating if needed 2022-03-10T03:12:51.227 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:51.235825 I | op-k8sutil: deployment "rook-ceph-mon-a" did not change, nothing to update 2022-03-10T03:12:51.227 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:51.235854 I | op-mon: waiting for mon quorum with [a b c] 2022-03-10T03:12:51.797 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:51.806445 I | op-mon: mons running: [a b c] 2022-03-10T03:12:52.221 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:52.229778 I | op-mon: Monitors in quorum: [a b c] 2022-03-10T03:12:52.228 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:52.237680 I | op-mon: deployment for mon rook-ceph-mon-b already exists. updating if needed 2022-03-10T03:12:52.240 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:52.248968 I | op-k8sutil: deployment "rook-ceph-mon-b" did not change, nothing to update 2022-03-10T03:12:52.240 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:52.248993 I | op-mon: waiting for mon quorum with [a b c] 2022-03-10T03:12:52.399 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:52.408236 I | op-mon: mons running: [a b c] 2022-03-10T03:12:52.824 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:52.833485 I | op-mon: Monitors in quorum: [a b c] 2022-03-10T03:12:52.834 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:52.843625 I | op-mon: deployment for mon rook-ceph-mon-c already exists. updating if needed 2022-03-10T03:12:52.841 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:52.850766 I | op-k8sutil: deployment "rook-ceph-mon-c" did not change, nothing to update 2022-03-10T03:12:52.842 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:52.850783 I | op-mon: waiting for mon quorum with [a b c] 2022-03-10T03:12:52.997 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:53.006269 I | op-mon: mons running: [a b c] 2022-03-10T03:12:53.433 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:53.442098 I | op-mon: Monitors in quorum: [a b c] 2022-03-10T03:12:53.433 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:53.442132 I | op-mon: mons created: 3 2022-03-10T03:12:53.859 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:53.867867 I | op-mon: waiting for mon quorum with [a b c] 2022-03-10T03:12:53.888 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:53.897377 I | op-mon: mons running: [a b c] 2022-03-10T03:12:54.298 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:54.307174 I | op-mon: Monitors in quorum: [a b c] 2022-03-10T03:12:54.299 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:54.307222 I | cephclient: getting or creating ceph auth key "client.csi-rbd-provisioner" 2022-03-10T03:12:54.717 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:54.725581 I | cephclient: getting or creating ceph auth key "client.csi-rbd-node" 2022-03-10T03:12:55.147 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:55.155715 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-provisioner" 2022-03-10T03:12:55.631 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:55.639547 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-node" 2022-03-10T03:12:56.107 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:56.115573 I | ceph-csi: created kubernetes csi secrets for cluster "rook-ceph" 2022-03-10T03:12:56.107 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:56.115606 I | cephclient: getting or creating ceph auth key "client.crash" 2022-03-10T03:12:56.556 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:56.564189 I | ceph-crashcollector-controller: created kubernetes crash collector secret for cluster "rook-ceph" 2022-03-10T03:12:57.000 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:57.008812 I | cephclient: successfully enabled msgr2 protocol 2022-03-10T03:12:57.001 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:57.008848 I | cephclient: create rbd-mirror bootstrap peer token "client.rbd-mirror-peer" 2022-03-10T03:12:57.001 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:57.008859 I | cephclient: getting or creating ceph auth key "client.rbd-mirror-peer" 2022-03-10T03:12:57.410 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:57.418498 I | cephclient: successfully created rbd-mirror bootstrap peer token for cluster "rook-ceph" 2022-03-10T03:12:57.451 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:57.460297 I | op-mgr: start running mgr 2022-03-10T03:12:57.452 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:57.460339 I | cephclient: getting or creating ceph auth key "mgr.a" 2022-03-10T03:12:57.813 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph osd stat -f json 2022-03-10T03:12:57.933 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:57.942189 I | op-mgr: deployment for mgr rook-ceph-mgr-a already exists. updating if needed 2022-03-10T03:12:57.942 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:57.950975 I | op-k8sutil: deployment "rook-ceph-mgr-a" did not change, nothing to update 2022-03-10T03:12:57.945 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:57.954150 I | op-mgr: setting services to point to mgr "a" 2022-03-10T03:12:58.069 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:58.069789 I | op-mgr: no need to update service "rook-ceph-mgr" 2022-03-10T03:12:58.070 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:58.069894 I | op-mgr: successful modules: balancer 2022-03-10T03:12:58.084 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:58.085132 I | op-osd: start running osds in namespace "rook-ceph" 2022-03-10T03:12:58.084 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:58.085178 I | op-osd: wait timeout for healthy OSDs during upgrade or restart is "10m0s" 2022-03-10T03:12:58.094 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:58.103226 I | op-osd: start provisioning the OSDs on PVCs, if needed 2022-03-10T03:12:58.269 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:58.277498 I | op-osd: creating 1 new PVCs for device set "smithi037-lv-1" 2022-03-10T03:12:58.395 INFO:teuthology.orchestra.run.smithi037.stdout: 2022-03-10T03:12:58.417 INFO:teuthology.orchestra.run.smithi037.stdout:{"epoch":2,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2022-03-10T03:12:58.417 INFO:tasks.rook: have 0/12 OSDs 2022-03-10T03:12:58.465 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:58.473348 I | op-osd: successfully provisioned PVC "smithi037-lv-1-data-08t564" 2022-03-10T03:12:58.465 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:58.473377 I | op-osd: creating 1 new PVCs for device set "smithi037-lv-2" 2022-03-10T03:12:58.664 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:58.671975 I | op-osd: successfully provisioned PVC "smithi037-lv-2-data-0hgjtd" 2022-03-10T03:12:58.664 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:58.672017 I | op-osd: creating 1 new PVCs for device set "smithi037-lv-3" 2022-03-10T03:12:58.862 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:58.870796 I | op-osd: successfully provisioned PVC "smithi037-lv-3-data-0z8qlf" 2022-03-10T03:12:58.863 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:58.870838 I | op-osd: creating 1 new PVCs for device set "smithi037-lv-4" 2022-03-10T03:12:59.063 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:59.071369 I | op-osd: successfully provisioned PVC "smithi037-lv-4-data-0pfq7x" 2022-03-10T03:12:59.063 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:59.071406 I | op-osd: creating 1 new PVCs for device set "smithi038-lv-1" 2022-03-10T03:12:59.278 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:59.286821 I | op-osd: successfully provisioned PVC "smithi038-lv-1-data-0bh7mw" 2022-03-10T03:12:59.279 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:59.286861 I | op-osd: creating 1 new PVCs for device set "smithi038-lv-2" 2022-03-10T03:12:59.345 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:59.353140 I | op-mgr: successful modules: mgr module(s) from the spec 2022-03-10T03:12:59.348 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:59.355121 I | op-mgr: successful modules: prometheus 2022-03-10T03:12:59.397 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:59.405215 I | op-mgr: successful modules: dashboard 2022-03-10T03:12:59.463 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:59.470937 I | op-osd: successfully provisioned PVC "smithi038-lv-2-data-0hvtxr" 2022-03-10T03:12:59.463 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:59.470973 I | op-osd: creating 1 new PVCs for device set "smithi038-lv-3" 2022-03-10T03:12:59.664 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:59.671949 I | op-osd: successfully provisioned PVC "smithi038-lv-3-data-0h4lqd" 2022-03-10T03:12:59.664 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:59.671987 I | op-osd: creating 1 new PVCs for device set "smithi038-lv-4" 2022-03-10T03:12:59.863 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:59.870856 I | op-osd: successfully provisioned PVC "smithi038-lv-4-data-0pzz62" 2022-03-10T03:12:59.863 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:12:59.870897 I | op-osd: creating 1 new PVCs for device set "smithi195-lv-1" 2022-03-10T03:13:00.086 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:13:00.094311 I | op-osd: successfully provisioned PVC "smithi195-lv-1-data-0jjrtk" 2022-03-10T03:13:00.086 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:13:00.094358 I | op-osd: creating 1 new PVCs for device set "smithi195-lv-2" 2022-03-10T03:13:00.263 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:13:00.271319 I | op-osd: successfully provisioned PVC "smithi195-lv-2-data-0cgctr" 2022-03-10T03:13:00.263 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:13:00.271354 I | op-osd: creating 1 new PVCs for device set "smithi195-lv-3" 2022-03-10T03:13:00.463 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:13:00.471526 I | op-osd: successfully provisioned PVC "smithi195-lv-3-data-0qs2g7" 2022-03-10T03:13:00.464 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:13:00.471559 I | op-osd: creating 1 new PVCs for device set "smithi195-lv-4" 2022-03-10T03:13:00.663 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:13:00.671208 I | op-osd: successfully provisioned PVC "smithi195-lv-4-data-0v9ftf" 2022-03-10T03:13:00.687 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:13:00.694810 I | op-osd: OSD will have its main bluestore block on "smithi037-lv-1-data-08t564" 2022-03-10T03:13:01.004 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:13:01.012613 I | op-mgr: successful modules: orchestrator modules 2022-03-10T03:13:01.086 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:13:01.093874 I | op-osd: started OSD provisioning job for PVC "smithi037-lv-1-data-08t564" 2022-03-10T03:13:01.086 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:13:01.093912 I | op-osd: OSD will have its main bluestore block on "smithi037-lv-2-data-0hgjtd" 2022-03-10T03:13:01.492 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:13:01.500245 I | op-osd: started OSD provisioning job for PVC "smithi037-lv-2-data-0hgjtd" 2022-03-10T03:13:01.492 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:13:01.500281 I | op-osd: OSD will have its main bluestore block on "smithi037-lv-3-data-0z8qlf" 2022-03-10T03:13:01.896 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:13:01.903797 I | op-osd: started OSD provisioning job for PVC "smithi037-lv-3-data-0z8qlf" 2022-03-10T03:13:01.896 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:13:01.903826 I | op-osd: OSD will have its main bluestore block on "smithi037-lv-4-data-0pfq7x" 2022-03-10T03:13:02.748 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:13:02.755794 I | op-osd: started OSD provisioning job for PVC "smithi037-lv-4-data-0pfq7x" 2022-03-10T03:13:02.748 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:13:02.755829 I | op-osd: OSD will have its main bluestore block on "smithi038-lv-1-data-0bh7mw" 2022-03-10T03:13:03.280 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:13:03.287484 I | op-osd: started OSD provisioning job for PVC "smithi038-lv-1-data-0bh7mw" 2022-03-10T03:13:03.280 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:13:03.287522 I | op-osd: OSD will have its main bluestore block on "smithi038-lv-2-data-0hvtxr" 2022-03-10T03:13:03.972 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:13:03.979285 I | op-osd: started OSD provisioning job for PVC "smithi038-lv-2-data-0hvtxr" 2022-03-10T03:13:03.972 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:13:03.979337 I | op-osd: OSD will have its main bluestore block on "smithi038-lv-3-data-0h4lqd" 2022-03-10T03:13:04.523 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:13:04.530753 I | op-osd: started OSD provisioning job for PVC "smithi038-lv-3-data-0h4lqd" 2022-03-10T03:13:04.524 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:13:04.530792 I | op-osd: OSD will have its main bluestore block on "smithi038-lv-4-data-0pzz62" 2022-03-10T03:13:04.712 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:13:04.720215 I | op-osd: started OSD provisioning job for PVC "smithi038-lv-4-data-0pzz62" 2022-03-10T03:13:04.713 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:13:04.720251 I | op-osd: OSD will have its main bluestore block on "smithi195-lv-1-data-0jjrtk" 2022-03-10T03:13:04.919 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:13:04.926960 I | op-osd: started OSD provisioning job for PVC "smithi195-lv-1-data-0jjrtk" 2022-03-10T03:13:04.919 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:13:04.926983 I | op-osd: OSD will have its main bluestore block on "smithi195-lv-2-data-0cgctr" 2022-03-10T03:13:05.345 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:13:05.352466 I | op-osd: started OSD provisioning job for PVC "smithi195-lv-2-data-0cgctr" 2022-03-10T03:13:05.345 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:13:05.352490 I | op-osd: OSD will have its main bluestore block on "smithi195-lv-3-data-0qs2g7" 2022-03-10T03:13:05.503 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:13:05.510396 I | op-osd: started OSD provisioning job for PVC "smithi195-lv-3-data-0qs2g7" 2022-03-10T03:13:05.642 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:13:05.510421 I | op-osd: OSD will have its main bluestore block on "smithi195-lv-4-data-0v9ftf" 2022-03-10T03:13:05.644 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:13:05.573264 I | op-osd: started OSD provisioning job for PVC "smithi195-lv-4-data-0v9ftf" 2022-03-10T03:13:05.644 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:13:05.573287 I | op-osd: start provisioning the OSDs on nodes, if needed 2022-03-10T03:13:05.645 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:13:05.573291 I | op-osd: no nodes are defined for configuring OSDs on raw devices 2022-03-10T03:13:05.681 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:13:05.687798 I | op-osd: OSD orchestration status for PVC smithi037-lv-1-data-08t564 is "starting" 2022-03-10T03:13:05.682 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:13:05.687840 I | op-osd: OSD orchestration status for PVC smithi037-lv-2-data-0hgjtd is "starting" 2022-03-10T03:13:05.682 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:13:05.687855 I | op-osd: OSD orchestration status for PVC smithi037-lv-3-data-0z8qlf is "starting" 2022-03-10T03:13:05.682 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:13:05.687875 I | op-osd: OSD orchestration status for PVC smithi037-lv-4-data-0pfq7x is "starting" 2022-03-10T03:13:05.682 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:13:05.687893 I | op-osd: OSD orchestration status for PVC smithi038-lv-1-data-0bh7mw is "starting" 2022-03-10T03:13:05.683 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:13:05.687905 I | op-osd: OSD orchestration status for PVC smithi038-lv-2-data-0hvtxr is "starting" 2022-03-10T03:13:05.683 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:13:05.687920 I | op-osd: OSD orchestration status for PVC smithi038-lv-3-data-0h4lqd is "starting" 2022-03-10T03:13:05.683 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:13:05.687935 I | op-osd: OSD orchestration status for PVC smithi038-lv-4-data-0pzz62 is "starting" 2022-03-10T03:13:05.683 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:13:05.687960 I | op-osd: OSD orchestration status for PVC smithi195-lv-1-data-0jjrtk is "starting" 2022-03-10T03:13:05.683 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:13:05.687986 I | op-osd: OSD orchestration status for PVC smithi195-lv-2-data-0cgctr is "starting" 2022-03-10T03:13:05.684 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:13:05.688007 I | op-osd: OSD orchestration status for PVC smithi195-lv-3-data-0qs2g7 is "starting" 2022-03-10T03:13:05.684 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:13:05.688029 I | op-osd: OSD orchestration status for PVC smithi195-lv-4-data-0v9ftf is "starting" 2022-03-10T03:13:08.419 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph osd stat -f json 2022-03-10T03:13:09.015 INFO:teuthology.orchestra.run.smithi037.stdout: 2022-03-10T03:13:09.032 INFO:teuthology.orchestra.run.smithi037.stdout:{"epoch":2,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2022-03-10T03:13:09.033 INFO:tasks.rook: have 0/12 OSDs 2022-03-10T03:13:17.839 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:13:17.846492 I | op-osd: OSD orchestration status for node smithi038-lv-1-data-0bh7mw is "orchestrating" 2022-03-10T03:13:18.638 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:13:18.645530 I | op-osd: OSD orchestration status for PVC smithi038-lv-1-data-0bh7mw is "orchestrating" 2022-03-10T03:13:19.034 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph osd stat -f json 2022-03-10T03:13:19.251 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:13:19.258424 I | op-osd: OSD orchestration status for node smithi037-lv-3-data-0z8qlf is "orchestrating" 2022-03-10T03:13:20.401 INFO:teuthology.orchestra.run.smithi037.stdout: 2022-03-10T03:13:20.420 INFO:teuthology.orchestra.run.smithi037.stdout:{"epoch":2,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2022-03-10T03:13:20.421 INFO:tasks.rook: have 0/12 OSDs 2022-03-10T03:13:20.647 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:13:20.654497 I | op-osd: OSD orchestration status for node smithi037-lv-2-data-0hgjtd is "orchestrating" 2022-03-10T03:13:21.099 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:13:21.090294 I | op-osd: OSD orchestration status for PVC smithi037-lv-3-data-0z8qlf is "orchestrating" 2022-03-10T03:13:21.258 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:13:21.264305 I | op-osd: OSD orchestration status for node smithi038-lv-4-data-0pzz62 is "orchestrating" 2022-03-10T03:13:21.662 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:13:21.669136 I | op-osd: OSD orchestration status for PVC smithi037-lv-2-data-0hgjtd is "orchestrating" 2022-03-10T03:13:22.162 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:13:22.168811 I | op-osd: OSD orchestration status for PVC smithi038-lv-4-data-0pzz62 is "orchestrating" 2022-03-10T03:13:23.452 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:13:23.458344 I | op-osd: OSD orchestration status for node smithi038-lv-3-data-0h4lqd is "orchestrating" 2022-03-10T03:13:23.452 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:13:23.458415 I | op-osd: OSD orchestration status for node smithi038-lv-2-data-0hvtxr is "orchestrating" 2022-03-10T03:13:24.335 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:13:24.341381 I | op-osd: OSD orchestration status for node smithi195-lv-3-data-0qs2g7 is "orchestrating" 2022-03-10T03:13:24.339 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:13:24.346362 I | op-osd: OSD orchestration status for PVC smithi038-lv-3-data-0h4lqd is "orchestrating" 2022-03-10T03:13:24.587 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:13:24.593368 I | op-osd: OSD orchestration status for PVC smithi038-lv-2-data-0hvtxr is "orchestrating" 2022-03-10T03:13:25.056 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:13:25.062621 I | op-osd: OSD orchestration status for PVC smithi195-lv-3-data-0qs2g7 is "orchestrating" 2022-03-10T03:13:25.735 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:13:25.740878 I | op-osd: OSD orchestration status for node smithi195-lv-1-data-0jjrtk is "orchestrating" 2022-03-10T03:13:25.735 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:13:25.740987 I | op-osd: OSD orchestration status for PVC smithi038-lv-1-data-0bh7mw is "completed" 2022-03-10T03:13:25.735 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:13:25.741011 I | op-osd: creating OSD 0 on PVC "smithi038-lv-1-data-0bh7mw" 2022-03-10T03:13:25.736 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:13:25.741032 I | op-osd: OSD will have its main bluestore block on "smithi038-lv-1-data-0bh7mw" 2022-03-10T03:13:26.043 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:13:26.050173 I | op-osd: OSD orchestration status for node smithi037-lv-1-data-08t564 is "orchestrating" 2022-03-10T03:13:26.693 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:13:26.698076 I | op-osd: OSD orchestration status for PVC smithi195-lv-1-data-0jjrtk is "orchestrating" 2022-03-10T03:13:27.226 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:13:27.232908 I | op-osd: OSD orchestration status for PVC smithi037-lv-1-data-08t564 is "orchestrating" 2022-03-10T03:13:27.258 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:13:27.264519 I | op-osd: OSD orchestration status for node smithi195-lv-2-data-0cgctr is "orchestrating" 2022-03-10T03:13:27.358 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:13:27.364978 I | op-osd: OSD orchestration status for node smithi195-lv-4-data-0v9ftf is "orchestrating" 2022-03-10T03:13:27.651 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:13:27.657973 I | op-osd: OSD orchestration status for node smithi037-lv-4-data-0pfq7x is "orchestrating" 2022-03-10T03:13:27.716 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:13:27.723061 I | op-osd: OSD orchestration status for PVC smithi195-lv-2-data-0cgctr is "orchestrating" 2022-03-10T03:13:27.808 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:13:27.814973 I | op-osd: OSD orchestration status for PVC smithi195-lv-4-data-0v9ftf is "orchestrating" 2022-03-10T03:13:28.134 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:13:28.141134 I | op-osd: OSD orchestration status for PVC smithi037-lv-4-data-0pfq7x is "orchestrating" 2022-03-10T03:13:29.892 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:13:29.897902 I | op-osd: OSD orchestration status for PVC smithi037-lv-3-data-0z8qlf is "completed" 2022-03-10T03:13:29.892 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:13:29.897929 I | op-osd: creating OSD 1 on PVC "smithi037-lv-3-data-0z8qlf" 2022-03-10T03:13:29.893 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:13:29.897941 I | op-osd: OSD will have its main bluestore block on "smithi037-lv-3-data-0z8qlf" 2022-03-10T03:13:30.422 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph osd stat -f json 2022-03-10T03:13:30.716 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:13:30.722094 I | op-osd: OSD orchestration status for PVC smithi037-lv-2-data-0hgjtd is "completed" 2022-03-10T03:13:30.717 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:13:30.722136 I | op-osd: creating OSD 2 on PVC "smithi037-lv-2-data-0hgjtd" 2022-03-10T03:13:30.717 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:13:30.722157 I | op-osd: OSD will have its main bluestore block on "smithi037-lv-2-data-0hgjtd" 2022-03-10T03:13:31.983 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:13:31.987811 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi037": the object has been modified; please apply your changes to the latest version and try again 2022-03-10T03:13:31.984 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:13:31.987886 I | op-osd: OSD orchestration status for PVC smithi038-lv-4-data-0pzz62 is "completed" 2022-03-10T03:13:31.984 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:13:31.987910 I | op-osd: creating OSD 3 on PVC "smithi038-lv-4-data-0pzz62" 2022-03-10T03:13:31.984 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:13:31.987926 I | op-osd: OSD will have its main bluestore block on "smithi038-lv-4-data-0pzz62" 2022-03-10T03:13:32.519 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:13:32.525224 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi037": the object has been modified; please apply your changes to the latest version and try again 2022-03-10T03:13:32.854 INFO:teuthology.orchestra.run.smithi037.stdout: 2022-03-10T03:13:32.875 INFO:teuthology.orchestra.run.smithi037.stdout:{"epoch":14,"num_osds":12,"num_up_osds":0,"osd_up_since":0,"num_in_osds":12,"osd_in_since":1646882011,"num_remapped_pgs":0} 2022-03-10T03:13:32.876 INFO:tasks.rook: have 0/12 OSDs 2022-03-10T03:13:32.910 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:13:32.916128 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi037": the object has been modified; please apply your changes to the latest version and try again 2022-03-10T03:13:34.611 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:13:34.609630 I | op-osd: OSD orchestration status for PVC smithi195-lv-3-data-0qs2g7 is "completed" 2022-03-10T03:13:34.611 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:13:34.609685 I | op-osd: creating OSD 6 on PVC "smithi195-lv-3-data-0qs2g7" 2022-03-10T03:13:34.611 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:13:34.609710 I | op-osd: OSD will have its main bluestore block on "smithi195-lv-3-data-0qs2g7" 2022-03-10T03:13:35.506 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:13:35.512189 I | op-osd: OSD orchestration status for PVC smithi038-lv-3-data-0h4lqd is "completed" 2022-03-10T03:13:35.507 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:13:35.512225 I | op-osd: creating OSD 4 on PVC "smithi038-lv-3-data-0h4lqd" 2022-03-10T03:13:35.507 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:13:35.512241 I | op-osd: OSD will have its main bluestore block on "smithi038-lv-3-data-0h4lqd" 2022-03-10T03:13:36.868 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:13:36.864909 I | op-osd: OSD orchestration status for PVC smithi038-lv-2-data-0hvtxr is "completed" 2022-03-10T03:13:36.868 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:13:36.864949 I | op-osd: creating OSD 5 on PVC "smithi038-lv-2-data-0hvtxr" 2022-03-10T03:13:36.869 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:13:36.864971 I | op-osd: OSD will have its main bluestore block on "smithi038-lv-2-data-0hvtxr" 2022-03-10T03:13:38.845 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:13:38.850793 I | op-osd: OSD orchestration status for PVC smithi195-lv-1-data-0jjrtk is "completed" 2022-03-10T03:13:38.846 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:13:38.850824 I | op-osd: creating OSD 7 on PVC "smithi195-lv-1-data-0jjrtk" 2022-03-10T03:13:38.846 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:13:38.850838 I | op-osd: OSD will have its main bluestore block on "smithi195-lv-1-data-0jjrtk" 2022-03-10T03:13:39.667 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:13:39.672578 I | op-osd: OSD orchestration status for PVC smithi195-lv-2-data-0cgctr is "completed" 2022-03-10T03:13:39.667 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:13:39.672617 I | op-osd: creating OSD 10 on PVC "smithi195-lv-2-data-0cgctr" 2022-03-10T03:13:39.668 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:13:39.672643 I | op-osd: OSD will have its main bluestore block on "smithi195-lv-2-data-0cgctr" 2022-03-10T03:13:40.505 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:13:40.510985 I | op-osd: OSD orchestration status for PVC smithi195-lv-4-data-0v9ftf is "completed" 2022-03-10T03:13:40.506 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:13:40.511010 I | op-osd: creating OSD 9 on PVC "smithi195-lv-4-data-0v9ftf" 2022-03-10T03:13:40.506 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:13:40.511021 I | op-osd: OSD will have its main bluestore block on "smithi195-lv-4-data-0v9ftf" 2022-03-10T03:13:41.576 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:13:41.581752 I | op-osd: OSD orchestration status for PVC smithi037-lv-1-data-08t564 is "completed" 2022-03-10T03:13:41.576 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:13:41.581778 I | op-osd: creating OSD 8 on PVC "smithi037-lv-1-data-08t564" 2022-03-10T03:13:41.577 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:13:41.581786 I | op-osd: OSD will have its main bluestore block on "smithi037-lv-1-data-08t564" 2022-03-10T03:13:42.878 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph osd stat -f json 2022-03-10T03:13:43.234 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:13:43.238794 I | op-osd: OSD orchestration status for PVC smithi037-lv-4-data-0pfq7x is "completed" 2022-03-10T03:13:43.235 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:13:43.238836 I | op-osd: creating OSD 11 on PVC "smithi037-lv-4-data-0pfq7x" 2022-03-10T03:13:43.235 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:13:43.238864 I | op-osd: OSD will have its main bluestore block on "smithi037-lv-4-data-0pfq7x" 2022-03-10T03:13:43.783 INFO:teuthology.orchestra.run.smithi037.stdout: 2022-03-10T03:13:43.806 INFO:teuthology.orchestra.run.smithi037.stdout:{"epoch":14,"num_osds":12,"num_up_osds":0,"osd_up_since":0,"num_in_osds":12,"osd_in_since":1646882011,"num_remapped_pgs":0} 2022-03-10T03:13:43.806 INFO:tasks.rook: have 0/12 OSDs 2022-03-10T03:13:46.011 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:13:46.016377 I | op-osd: finished running OSDs in namespace "rook-ceph" 2022-03-10T03:13:46.011 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:13:46.016412 I | ceph-cluster-controller: done reconciling ceph cluster in namespace "rook-ceph" 2022-03-10T03:13:53.807 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph osd stat -f json 2022-03-10T03:13:57.014 INFO:teuthology.orchestra.run.smithi037.stdout: 2022-03-10T03:13:57.040 INFO:teuthology.orchestra.run.smithi037.stdout:{"epoch":14,"num_osds":12,"num_up_osds":0,"osd_up_since":0,"num_in_osds":12,"osd_in_since":1646882011,"num_remapped_pgs":0} 2022-03-10T03:13:57.041 INFO:tasks.rook: have 0/12 OSDs 2022-03-10T03:14:07.043 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph osd stat -f json 2022-03-10T03:14:08.486 INFO:teuthology.orchestra.run.smithi037.stdout: 2022-03-10T03:14:08.511 INFO:teuthology.orchestra.run.smithi037.stdout:{"epoch":21,"num_osds":12,"num_up_osds":3,"osd_up_since":1646882047,"num_in_osds":12,"osd_in_since":1646882011,"num_remapped_pgs":0} 2022-03-10T03:14:08.511 INFO:tasks.rook: have 3/12 OSDs 2022-03-10T03:14:18.513 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph osd stat -f json 2022-03-10T03:14:19.079 INFO:teuthology.orchestra.run.smithi037.stdout: 2022-03-10T03:14:19.098 INFO:teuthology.orchestra.run.smithi037.stdout:{"epoch":29,"num_osds":12,"num_up_osds":12,"osd_up_since":1646882058,"num_in_osds":12,"osd_in_since":1646882011,"num_remapped_pgs":0} 2022-03-10T03:14:19.098 INFO:tasks.rook:Distributing ceph config and client.admin keyring 2022-03-10T03:14:19.099 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- cat /etc/ceph/ceph.conf 2022-03-10T03:14:19.293 INFO:teuthology.orchestra.run.smithi037.stdout:[global] 2022-03-10T03:14:19.294 INFO:teuthology.orchestra.run.smithi037.stdout:mon_host = 10.100.186.100:6789,10.111.132.151:6789,10.101.46.100:6789 2022-03-10T03:14:19.294 INFO:teuthology.orchestra.run.smithi037.stdout: 2022-03-10T03:14:19.294 INFO:teuthology.orchestra.run.smithi037.stdout:[client.admin] 2022-03-10T03:14:19.294 INFO:teuthology.orchestra.run.smithi037.stdout:keyring = /etc/ceph/keyring 2022-03-10T03:14:19.298 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- cat /etc/ceph/keyring 2022-03-10T03:14:19.493 INFO:teuthology.orchestra.run.smithi037.stdout:[client.admin] 2022-03-10T03:14:19.494 INFO:teuthology.orchestra.run.smithi037.stdout:key = AQDKayliVjTbCBAAh0DAdnk9TNPlOesueI3SNQ== 2022-03-10T03:14:19.497 DEBUG:teuthology.orchestra.run.smithi037:> sudo mkdir -p /etc/ceph 2022-03-10T03:14:19.509 DEBUG:teuthology.orchestra.run.smithi038:> sudo mkdir -p /etc/ceph 2022-03-10T03:14:19.525 DEBUG:teuthology.orchestra.run.smithi195:> sudo mkdir -p /etc/ceph 2022-03-10T03:14:19.540 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2022-03-10T03:14:19.540 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd of=/etc/ceph/ceph.conf 2022-03-10T03:14:19.560 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2022-03-10T03:14:19.561 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd of=/etc/ceph/keyring 2022-03-10T03:14:19.571 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2022-03-10T03:14:19.571 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/etc/ceph/ceph.conf 2022-03-10T03:14:19.581 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2022-03-10T03:14:19.582 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/etc/ceph/keyring 2022-03-10T03:14:19.592 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2022-03-10T03:14:19.593 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd of=/etc/ceph/ceph.conf 2022-03-10T03:14:19.611 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2022-03-10T03:14:19.611 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd of=/etc/ceph/keyring 2022-03-10T03:14:19.626 INFO:tasks.rook:Setting up client nodes... 2022-03-10T03:14:19.627 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph auth get-or-create client.a mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2022-03-10T03:14:20.472 INFO:teuthology.orchestra.run.smithi037.stdout:[client.a] 2022-03-10T03:14:20.472 INFO:teuthology.orchestra.run.smithi037.stdout: key = AQAMbSlilkWGFhAASJDB+/2TlRLpenM3znOl3g== 2022-03-10T03:14:20.495 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2022-03-10T03:14:20.496 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd of=/etc/ceph/ceph.client.a.keyring 2022-03-10T03:14:20.496 DEBUG:teuthology.orchestra.run.smithi037:> sudo chmod 0644 /etc/ceph/ceph.client.a.keyring 2022-03-10T03:14:20.512 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph auth get-or-create client.b mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2022-03-10T03:14:21.273 INFO:teuthology.orchestra.run.smithi037.stdout:[client.b] 2022-03-10T03:14:21.273 INFO:teuthology.orchestra.run.smithi037.stdout: key = AQANbSlidgegChAAQ/de8sQBX1dcX7XA0luoaA== 2022-03-10T03:14:21.296 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2022-03-10T03:14:21.297 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/etc/ceph/ceph.client.b.keyring 2022-03-10T03:14:21.297 DEBUG:teuthology.orchestra.run.smithi038:> sudo chmod 0644 /etc/ceph/ceph.client.b.keyring 2022-03-10T03:14:21.316 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph auth get-or-create client.c mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2022-03-10T03:14:22.064 INFO:teuthology.orchestra.run.smithi037.stdout:[client.c] 2022-03-10T03:14:22.065 INFO:teuthology.orchestra.run.smithi037.stdout: key = AQANbSliF6kjOxAACBjey9VETeLHm/JrS5PWIg== 2022-03-10T03:14:22.088 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2022-03-10T03:14:22.089 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd of=/etc/ceph/ceph.client.c.keyring 2022-03-10T03:14:22.089 DEBUG:teuthology.orchestra.run.smithi195:> sudo chmod 0644 /etc/ceph/ceph.client.c.keyring 2022-03-10T03:14:22.115 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph osd dump --format=json 2022-03-10T03:14:22.661 INFO:teuthology.orchestra.run.smithi037.stdout: 2022-03-10T03:14:22.661 INFO:teuthology.orchestra.run.smithi037.stdout:{"epoch":30,"fsid":"2d9bcb02-6465-4ca2-9864-d3ce67476d09","created":"2022-03-10T03:10:27.753645+0000","modified":"2022-03-10T03:14:20.069737+0000","last_up_change":"2022-03-10T03:14:18.651315+0000","last_in_change":"2022-03-10T03:13:31.679040+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":12,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-03-10T03:14:09.793178+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"on","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"15f6bfc0-820b-422b-87b3-2caff2f53532","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.34.79:6800","nonce":1705037372},{"type":"v1","addr":"10.249.34.79:6801","nonce":1705037372}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.34.79:6802","nonce":1705037372},{"type":"v1","addr":"10.249.34.79:6803","nonce":1705037372}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.34.79:6806","nonce":1705037372},{"type":"v1","addr":"10.249.34.79:6807","nonce":1705037372}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.34.79:6804","nonce":1705037372},{"type":"v1","addr":"10.249.34.79:6805","nonce":1705037372}]},"public_addr":"10.249.34.79:6801/1705037372","cluster_addr":"10.249.34.79:6803/1705037372","heartbeat_back_addr":"10.249.34.79:6807/1705037372","heartbeat_front_addr":"10.249.34.79:6805/1705037372","state":["exists","up"]},{"osd":1,"uuid":"eeb3df13-66d8-4237-b42c-f41a4008730f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.32.75:6800","nonce":2396434981},{"type":"v1","addr":"10.249.32.75:6801","nonce":2396434981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.32.75:6802","nonce":2396434981},{"type":"v1","addr":"10.249.32.75:6803","nonce":2396434981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.32.75:6806","nonce":2396434981},{"type":"v1","addr":"10.249.32.75:6807","nonce":2396434981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.32.75:6804","nonce":2396434981},{"type":"v1","addr":"10.249.32.75:6805","nonce":2396434981}]},"public_addr":"10.249.32.75:6801/2396434981","cluster_addr":"10.249.32.75:6803/2396434981","heartbeat_back_addr":"10.249.32.75:6807/2396434981","heartbeat_front_addr":"10.249.32.75:6805/2396434981","state":["exists","up"]},{"osd":2,"uuid":"bf0df36b-48ee-4916-8267-de34c4981e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.32.76:6800","nonce":1980726265},{"type":"v1","addr":"10.249.32.76:6801","nonce":1980726265}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.32.76:6802","nonce":1980726265},{"type":"v1","addr":"10.249.32.76:6803","nonce":1980726265}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.32.76:6806","nonce":1980726265},{"type":"v1","addr":"10.249.32.76:6807","nonce":1980726265}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.32.76:6804","nonce":1980726265},{"type":"v1","addr":"10.249.32.76:6805","nonce":1980726265}]},"public_addr":"10.249.32.76:6801/1980726265","cluster_addr":"10.249.32.76:6803/1980726265","heartbeat_back_addr":"10.249.32.76:6807/1980726265","heartbeat_front_addr":"10.249.32.76:6805/1980726265","state":["exists","up"]},{"osd":3,"uuid":"109faadf-dfc8-44fb-b632-a56ca8b73443","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.34.80:6800","nonce":1532700234},{"type":"v1","addr":"10.249.34.80:6801","nonce":1532700234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.34.80:6802","nonce":1532700234},{"type":"v1","addr":"10.249.34.80:6803","nonce":1532700234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.34.80:6806","nonce":1532700234},{"type":"v1","addr":"10.249.34.80:6807","nonce":1532700234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.34.80:6804","nonce":1532700234},{"type":"v1","addr":"10.249.34.80:6805","nonce":1532700234}]},"public_addr":"10.249.34.80:6801/1532700234","cluster_addr":"10.249.34.80:6803/1532700234","heartbeat_back_addr":"10.249.34.80:6807/1532700234","heartbeat_front_addr":"10.249.34.80:6805/1532700234","state":["exists","up"]},{"osd":4,"uuid":"06149299-f29d-4f1a-b8ca-b4c274e41558","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.34.81:6800","nonce":3630779696},{"type":"v1","addr":"10.249.34.81:6801","nonce":3630779696}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.34.81:6802","nonce":3630779696},{"type":"v1","addr":"10.249.34.81:6803","nonce":3630779696}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.34.81:6806","nonce":3630779696},{"type":"v1","addr":"10.249.34.81:6807","nonce":3630779696}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.34.81:6804","nonce":3630779696},{"type":"v1","addr":"10.249.34.81:6805","nonce":3630779696}]},"public_addr":"10.249.34.81:6801/3630779696","cluster_addr":"10.249.34.81:6803/3630779696","heartbeat_back_addr":"10.249.34.81:6807/3630779696","heartbeat_front_addr":"10.249.34.81:6805/3630779696","state":["exists","up"]},{"osd":5,"uuid":"772999a7-ce8c-492e-b4a5-75e7ecd77b95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.34.82:6800","nonce":3725102274},{"type":"v1","addr":"10.249.34.82:6801","nonce":3725102274}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.34.82:6802","nonce":3725102274},{"type":"v1","addr":"10.249.34.82:6803","nonce":3725102274}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.34.82:6806","nonce":3725102274},{"type":"v1","addr":"10.249.34.82:6807","nonce":3725102274}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.34.82:6804","nonce":3725102274},{"type":"v1","addr":"10.249.34.82:6805","nonce":3725102274}]},"public_addr":"10.249.34.82:6801/3725102274","cluster_addr":"10.249.34.82:6803/3725102274","heartbeat_back_addr":"10.249.34.82:6807/3725102274","heartbeat_front_addr":"10.249.34.82:6805/3725102274","state":["exists","up"]},{"osd":6,"uuid":"42b0dfd3-0e2a-43fb-b166-5a7f0b57ade4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.36.140:6800","nonce":1429719811},{"type":"v1","addr":"10.249.36.140:6801","nonce":1429719811}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.36.140:6802","nonce":1429719811},{"type":"v1","addr":"10.249.36.140:6803","nonce":1429719811}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.36.140:6806","nonce":1429719811},{"type":"v1","addr":"10.249.36.140:6807","nonce":1429719811}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.36.140:6804","nonce":1429719811},{"type":"v1","addr":"10.249.36.140:6805","nonce":1429719811}]},"public_addr":"10.249.36.140:6801/1429719811","cluster_addr":"10.249.36.140:6803/1429719811","heartbeat_back_addr":"10.249.36.140:6807/1429719811","heartbeat_front_addr":"10.249.36.140:6805/1429719811","state":["exists","up"]},{"osd":7,"uuid":"aa7918fa-b0fe-4cce-86e0-b825160ccbab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.36.141:6800","nonce":3582135998},{"type":"v1","addr":"10.249.36.141:6801","nonce":3582135998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.36.141:6802","nonce":3582135998},{"type":"v1","addr":"10.249.36.141:6803","nonce":3582135998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.36.141:6806","nonce":3582135998},{"type":"v1","addr":"10.249.36.141:6807","nonce":3582135998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.36.141:6804","nonce":3582135998},{"type":"v1","addr":"10.249.36.141:6805","nonce":3582135998}]},"public_addr":"10.249.36.141:6801/3582135998","cluster_addr":"10.249.36.141:6803/3582135998","heartbeat_back_addr":"10.249.36.141:6807/3582135998","heartbeat_front_addr":"10.249.36.141:6805/3582135998","state":["exists","up"]},{"osd":8,"uuid":"8c3caec1-8e2b-4e29-b51e-7f0fb16b5804","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":29,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.32.78:6800","nonce":2902739700},{"type":"v1","addr":"10.249.32.78:6801","nonce":2902739700}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.32.78:6802","nonce":2902739700},{"type":"v1","addr":"10.249.32.78:6803","nonce":2902739700}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.32.78:6806","nonce":2902739700},{"type":"v1","addr":"10.249.32.78:6807","nonce":2902739700}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.32.78:6804","nonce":2902739700},{"type":"v1","addr":"10.249.32.78:6805","nonce":2902739700}]},"public_addr":"10.249.32.78:6801/2902739700","cluster_addr":"10.249.32.78:6803/2902739700","heartbeat_back_addr":"10.249.32.78:6807/2902739700","heartbeat_front_addr":"10.249.32.78:6805/2902739700","state":["exists","up"]},{"osd":9,"uuid":"7ec16300-d058-489d-b341-80c6f6375aa6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.36.143:6800","nonce":3569720756},{"type":"v1","addr":"10.249.36.143:6801","nonce":3569720756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.36.143:6802","nonce":3569720756},{"type":"v1","addr":"10.249.36.143:6803","nonce":3569720756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.36.143:6806","nonce":3569720756},{"type":"v1","addr":"10.249.36.143:6807","nonce":3569720756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.36.143:6804","nonce":3569720756},{"type":"v1","addr":"10.249.36.143:6805","nonce":3569720756}]},"public_addr":"10.249.36.143:6801/3569720756","cluster_addr":"10.249.36.143:6803/3569720756","heartbeat_back_addr":"10.249.36.143:6807/3569720756","heartbeat_front_addr":"10.249.36.143:6805/3569720756","state":["exists","up"]},{"osd":10,"uuid":"658b5090-fc93-4908-8889-f7b8ad1d5f46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.36.142:6800","nonce":2631742787},{"type":"v1","addr":"10.249.36.142:6801","nonce":2631742787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.36.142:6802","nonce":2631742787},{"type":"v1","addr":"10.249.36.142:6803","nonce":2631742787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.36.142:6806","nonce":2631742787},{"type":"v1","addr":"10.249.36.142:6807","nonce":2631742787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.36.142:6804","nonce":2631742787},{"type":"v1","addr":"10.249.36.142:6805","nonce":2631742787}]},"public_addr":"10.249.36.142:6801/2631742787","cluster_addr":"10.249.36.142:6803/2631742787","heartbeat_back_addr":"10.249.36.142:6807/2631742787","heartbeat_front_addr":"10.249.36.142:6805/2631742787","state":["exists","up"]},{"osd":11,"uuid":"9c3326a4-7ed4-49ed-937a-5c7a56125b33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":29,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.32.79:6800","nonce":2227166999},{"type":"v1","addr":"10.249.32.79:6801","nonce":2227166999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.32.79:6802","nonce":2227166999},{"type":"v1","addr":"10.249.32.79:6803","nonce":2227166999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.32.79:6806","nonce":2227166999},{"type":"v1","addr":"10.249.32.79:6807","nonce":2227166999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.32.79:6804","nonce":2227166999},{"type":"v1","addr":"10.249.32.79:6805","nonce":2227166999}]},"public_addr":"10.249.32.79:6801/2227166999","cluster_addr":"10.249.32.79:6803/2227166999","heartbeat_back_addr":"10.249.32.79:6807/2227166999","heartbeat_front_addr":"10.249.32.79:6805/2227166999","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-10T03:13:59.639154+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-10T03:14:08.444717+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-10T03:14:10.742204+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-10T03:14:03.219877+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-10T03:14:07.901081+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-10T03:14:09.584727+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-10T03:14:04.256752+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-10T03:14:10.209048+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-10T03:14:17.102473+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-10T03:14:10.182386+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-10T03:14:07.310478+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-10T03:14:17.200510+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.249.32.69:0/2108578815":"2022-03-11T03:12:22.565538+0000","10.249.32.69:6800/2959216379":"2022-03-11T03:12:22.565538+0000","10.249.32.69:6801/2959216379":"2022-03-11T03:12:22.565538+0000","10.249.32.69:0/1440386292":"2022-03-11T03:12:22.565538+0000","10.249.32.69:0/2380130538":"2022-03-11T03:12:22.565538+0000"},"erasure_code_profiles":{"default":{"k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-03-10T03:14:22.680 INFO:tasks.rook.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2022-03-10T03:14:09.793178+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'on', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '28', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2022-03-10T03:14:22.681 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph osd pool get .mgr pg_num 2022-03-10T03:14:23.218 INFO:teuthology.orchestra.run.smithi037.stdout:pg_num: 1 2022-03-10T03:14:23.241 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2022-03-10T03:14:23.242 INFO:tasks.rook.ceph_manager.ceph:waiting for mgr available 2022-03-10T03:14:23.242 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph mgr dump --format=json 2022-03-10T03:14:23.812 INFO:teuthology.orchestra.run.smithi037.stdout: 2022-03-10T03:14:23.828 INFO:teuthology.orchestra.run.smithi037.stdout:{"epoch":12,"active_gid":24158,"active_name":"a","active_addrs":{"addrvec":[{"type":"v2","addr":"10.249.32.69:6800","nonce":816458221},{"type":"v1","addr":"10.249.32.69:6801","nonce":816458221}]},"active_addr":"10.249.32.69:6801/816458221","active_change":"2022-03-10T03:12:22.600501+0000","active_mgr_features":4540138303579357183,"available":true,"standbys":[],"modules":["iostat","nfs","prometheus","restful","rook"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:6.7.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"0.0.0.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"0.0.0.0","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"prometheus":"http://10.249.32.69:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":2,"active_clients":[{"addrvec":[{"type":"v2","addr":"10.249.32.69:0","nonce":1991266620}]},{"addrvec":[{"type":"v2","addr":"10.249.32.69:0","nonce":1450864786}]},{"addrvec":[{"type":"v2","addr":"10.249.32.69:0","nonce":3919258737}]},{"addrvec":[{"type":"v2","addr":"10.249.32.69:0","nonce":668094100}]}]}} 2022-03-10T03:14:23.832 INFO:tasks.rook.ceph_manager.ceph:mgr available! 2022-03-10T03:14:23.832 INFO:tasks.rook.ceph_manager.ceph:waiting for all up 2022-03-10T03:14:23.832 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph osd dump --format=json 2022-03-10T03:14:24.426 INFO:teuthology.orchestra.run.smithi037.stdout: 2022-03-10T03:14:24.426 INFO:teuthology.orchestra.run.smithi037.stdout:{"epoch":30,"fsid":"2d9bcb02-6465-4ca2-9864-d3ce67476d09","created":"2022-03-10T03:10:27.753645+0000","modified":"2022-03-10T03:14:20.069737+0000","last_up_change":"2022-03-10T03:14:18.651315+0000","last_in_change":"2022-03-10T03:13:31.679040+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":12,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-03-10T03:14:09.793178+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"on","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"15f6bfc0-820b-422b-87b3-2caff2f53532","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.34.79:6800","nonce":1705037372},{"type":"v1","addr":"10.249.34.79:6801","nonce":1705037372}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.34.79:6802","nonce":1705037372},{"type":"v1","addr":"10.249.34.79:6803","nonce":1705037372}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.34.79:6806","nonce":1705037372},{"type":"v1","addr":"10.249.34.79:6807","nonce":1705037372}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.34.79:6804","nonce":1705037372},{"type":"v1","addr":"10.249.34.79:6805","nonce":1705037372}]},"public_addr":"10.249.34.79:6801/1705037372","cluster_addr":"10.249.34.79:6803/1705037372","heartbeat_back_addr":"10.249.34.79:6807/1705037372","heartbeat_front_addr":"10.249.34.79:6805/1705037372","state":["exists","up"]},{"osd":1,"uuid":"eeb3df13-66d8-4237-b42c-f41a4008730f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.32.75:6800","nonce":2396434981},{"type":"v1","addr":"10.249.32.75:6801","nonce":2396434981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.32.75:6802","nonce":2396434981},{"type":"v1","addr":"10.249.32.75:6803","nonce":2396434981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.32.75:6806","nonce":2396434981},{"type":"v1","addr":"10.249.32.75:6807","nonce":2396434981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.32.75:6804","nonce":2396434981},{"type":"v1","addr":"10.249.32.75:6805","nonce":2396434981}]},"public_addr":"10.249.32.75:6801/2396434981","cluster_addr":"10.249.32.75:6803/2396434981","heartbeat_back_addr":"10.249.32.75:6807/2396434981","heartbeat_front_addr":"10.249.32.75:6805/2396434981","state":["exists","up"]},{"osd":2,"uuid":"bf0df36b-48ee-4916-8267-de34c4981e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.32.76:6800","nonce":1980726265},{"type":"v1","addr":"10.249.32.76:6801","nonce":1980726265}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.32.76:6802","nonce":1980726265},{"type":"v1","addr":"10.249.32.76:6803","nonce":1980726265}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.32.76:6806","nonce":1980726265},{"type":"v1","addr":"10.249.32.76:6807","nonce":1980726265}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.32.76:6804","nonce":1980726265},{"type":"v1","addr":"10.249.32.76:6805","nonce":1980726265}]},"public_addr":"10.249.32.76:6801/1980726265","cluster_addr":"10.249.32.76:6803/1980726265","heartbeat_back_addr":"10.249.32.76:6807/1980726265","heartbeat_front_addr":"10.249.32.76:6805/1980726265","state":["exists","up"]},{"osd":3,"uuid":"109faadf-dfc8-44fb-b632-a56ca8b73443","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.34.80:6800","nonce":1532700234},{"type":"v1","addr":"10.249.34.80:6801","nonce":1532700234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.34.80:6802","nonce":1532700234},{"type":"v1","addr":"10.249.34.80:6803","nonce":1532700234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.34.80:6806","nonce":1532700234},{"type":"v1","addr":"10.249.34.80:6807","nonce":1532700234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.34.80:6804","nonce":1532700234},{"type":"v1","addr":"10.249.34.80:6805","nonce":1532700234}]},"public_addr":"10.249.34.80:6801/1532700234","cluster_addr":"10.249.34.80:6803/1532700234","heartbeat_back_addr":"10.249.34.80:6807/1532700234","heartbeat_front_addr":"10.249.34.80:6805/1532700234","state":["exists","up"]},{"osd":4,"uuid":"06149299-f29d-4f1a-b8ca-b4c274e41558","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.34.81:6800","nonce":3630779696},{"type":"v1","addr":"10.249.34.81:6801","nonce":3630779696}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.34.81:6802","nonce":3630779696},{"type":"v1","addr":"10.249.34.81:6803","nonce":3630779696}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.34.81:6806","nonce":3630779696},{"type":"v1","addr":"10.249.34.81:6807","nonce":3630779696}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.34.81:6804","nonce":3630779696},{"type":"v1","addr":"10.249.34.81:6805","nonce":3630779696}]},"public_addr":"10.249.34.81:6801/3630779696","cluster_addr":"10.249.34.81:6803/3630779696","heartbeat_back_addr":"10.249.34.81:6807/3630779696","heartbeat_front_addr":"10.249.34.81:6805/3630779696","state":["exists","up"]},{"osd":5,"uuid":"772999a7-ce8c-492e-b4a5-75e7ecd77b95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.34.82:6800","nonce":3725102274},{"type":"v1","addr":"10.249.34.82:6801","nonce":3725102274}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.34.82:6802","nonce":3725102274},{"type":"v1","addr":"10.249.34.82:6803","nonce":3725102274}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.34.82:6806","nonce":3725102274},{"type":"v1","addr":"10.249.34.82:6807","nonce":3725102274}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.34.82:6804","nonce":3725102274},{"type":"v1","addr":"10.249.34.82:6805","nonce":3725102274}]},"public_addr":"10.249.34.82:6801/3725102274","cluster_addr":"10.249.34.82:6803/3725102274","heartbeat_back_addr":"10.249.34.82:6807/3725102274","heartbeat_front_addr":"10.249.34.82:6805/3725102274","state":["exists","up"]},{"osd":6,"uuid":"42b0dfd3-0e2a-43fb-b166-5a7f0b57ade4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.36.140:6800","nonce":1429719811},{"type":"v1","addr":"10.249.36.140:6801","nonce":1429719811}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.36.140:6802","nonce":1429719811},{"type":"v1","addr":"10.249.36.140:6803","nonce":1429719811}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.36.140:6806","nonce":1429719811},{"type":"v1","addr":"10.249.36.140:6807","nonce":1429719811}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.36.140:6804","nonce":1429719811},{"type":"v1","addr":"10.249.36.140:6805","nonce":1429719811}]},"public_addr":"10.249.36.140:6801/1429719811","cluster_addr":"10.249.36.140:6803/1429719811","heartbeat_back_addr":"10.249.36.140:6807/1429719811","heartbeat_front_addr":"10.249.36.140:6805/1429719811","state":["exists","up"]},{"osd":7,"uuid":"aa7918fa-b0fe-4cce-86e0-b825160ccbab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.36.141:6800","nonce":3582135998},{"type":"v1","addr":"10.249.36.141:6801","nonce":3582135998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.36.141:6802","nonce":3582135998},{"type":"v1","addr":"10.249.36.141:6803","nonce":3582135998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.36.141:6806","nonce":3582135998},{"type":"v1","addr":"10.249.36.141:6807","nonce":3582135998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.36.141:6804","nonce":3582135998},{"type":"v1","addr":"10.249.36.141:6805","nonce":3582135998}]},"public_addr":"10.249.36.141:6801/3582135998","cluster_addr":"10.249.36.141:6803/3582135998","heartbeat_back_addr":"10.249.36.141:6807/3582135998","heartbeat_front_addr":"10.249.36.141:6805/3582135998","state":["exists","up"]},{"osd":8,"uuid":"8c3caec1-8e2b-4e29-b51e-7f0fb16b5804","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":29,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.32.78:6800","nonce":2902739700},{"type":"v1","addr":"10.249.32.78:6801","nonce":2902739700}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.32.78:6802","nonce":2902739700},{"type":"v1","addr":"10.249.32.78:6803","nonce":2902739700}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.32.78:6806","nonce":2902739700},{"type":"v1","addr":"10.249.32.78:6807","nonce":2902739700}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.32.78:6804","nonce":2902739700},{"type":"v1","addr":"10.249.32.78:6805","nonce":2902739700}]},"public_addr":"10.249.32.78:6801/2902739700","cluster_addr":"10.249.32.78:6803/2902739700","heartbeat_back_addr":"10.249.32.78:6807/2902739700","heartbeat_front_addr":"10.249.32.78:6805/2902739700","state":["exists","up"]},{"osd":9,"uuid":"7ec16300-d058-489d-b341-80c6f6375aa6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.36.143:6800","nonce":3569720756},{"type":"v1","addr":"10.249.36.143:6801","nonce":3569720756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.36.143:6802","nonce":3569720756},{"type":"v1","addr":"10.249.36.143:6803","nonce":3569720756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.36.143:6806","nonce":3569720756},{"type":"v1","addr":"10.249.36.143:6807","nonce":3569720756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.36.143:6804","nonce":3569720756},{"type":"v1","addr":"10.249.36.143:6805","nonce":3569720756}]},"public_addr":"10.249.36.143:6801/3569720756","cluster_addr":"10.249.36.143:6803/3569720756","heartbeat_back_addr":"10.249.36.143:6807/3569720756","heartbeat_front_addr":"10.249.36.143:6805/3569720756","state":["exists","up"]},{"osd":10,"uuid":"658b5090-fc93-4908-8889-f7b8ad1d5f46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.36.142:6800","nonce":2631742787},{"type":"v1","addr":"10.249.36.142:6801","nonce":2631742787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.36.142:6802","nonce":2631742787},{"type":"v1","addr":"10.249.36.142:6803","nonce":2631742787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.36.142:6806","nonce":2631742787},{"type":"v1","addr":"10.249.36.142:6807","nonce":2631742787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.36.142:6804","nonce":2631742787},{"type":"v1","addr":"10.249.36.142:6805","nonce":2631742787}]},"public_addr":"10.249.36.142:6801/2631742787","cluster_addr":"10.249.36.142:6803/2631742787","heartbeat_back_addr":"10.249.36.142:6807/2631742787","heartbeat_front_addr":"10.249.36.142:6805/2631742787","state":["exists","up"]},{"osd":11,"uuid":"9c3326a4-7ed4-49ed-937a-5c7a56125b33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":29,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.32.79:6800","nonce":2227166999},{"type":"v1","addr":"10.249.32.79:6801","nonce":2227166999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.32.79:6802","nonce":2227166999},{"type":"v1","addr":"10.249.32.79:6803","nonce":2227166999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.32.79:6806","nonce":2227166999},{"type":"v1","addr":"10.249.32.79:6807","nonce":2227166999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.32.79:6804","nonce":2227166999},{"type":"v1","addr":"10.249.32.79:6805","nonce":2227166999}]},"public_addr":"10.249.32.79:6801/2227166999","cluster_addr":"10.249.32.79:6803/2227166999","heartbeat_back_addr":"10.249.32.79:6807/2227166999","heartbeat_front_addr":"10.249.32.79:6805/2227166999","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-10T03:13:59.639154+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-10T03:14:08.444717+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-10T03:14:10.742204+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-10T03:14:03.219877+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-10T03:14:07.901081+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-10T03:14:09.584727+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-10T03:14:04.256752+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-10T03:14:10.209048+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-10T03:14:17.102473+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-10T03:14:10.182386+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-10T03:14:07.310478+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-10T03:14:17.200510+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.249.32.69:0/2108578815":"2022-03-11T03:12:22.565538+0000","10.249.32.69:6800/2959216379":"2022-03-11T03:12:22.565538+0000","10.249.32.69:6801/2959216379":"2022-03-11T03:12:22.565538+0000","10.249.32.69:0/1440386292":"2022-03-11T03:12:22.565538+0000","10.249.32.69:0/2380130538":"2022-03-11T03:12:22.565538+0000"},"erasure_code_profiles":{"default":{"k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-03-10T03:14:24.447 INFO:tasks.rook.ceph_manager.ceph:all up! 2022-03-10T03:14:24.448 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph osd dump --format=json 2022-03-10T03:14:24.996 INFO:teuthology.orchestra.run.smithi037.stdout: 2022-03-10T03:14:24.997 INFO:teuthology.orchestra.run.smithi037.stdout:{"epoch":30,"fsid":"2d9bcb02-6465-4ca2-9864-d3ce67476d09","created":"2022-03-10T03:10:27.753645+0000","modified":"2022-03-10T03:14:20.069737+0000","last_up_change":"2022-03-10T03:14:18.651315+0000","last_in_change":"2022-03-10T03:13:31.679040+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":12,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-03-10T03:14:09.793178+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"on","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"15f6bfc0-820b-422b-87b3-2caff2f53532","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.34.79:6800","nonce":1705037372},{"type":"v1","addr":"10.249.34.79:6801","nonce":1705037372}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.34.79:6802","nonce":1705037372},{"type":"v1","addr":"10.249.34.79:6803","nonce":1705037372}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.34.79:6806","nonce":1705037372},{"type":"v1","addr":"10.249.34.79:6807","nonce":1705037372}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.34.79:6804","nonce":1705037372},{"type":"v1","addr":"10.249.34.79:6805","nonce":1705037372}]},"public_addr":"10.249.34.79:6801/1705037372","cluster_addr":"10.249.34.79:6803/1705037372","heartbeat_back_addr":"10.249.34.79:6807/1705037372","heartbeat_front_addr":"10.249.34.79:6805/1705037372","state":["exists","up"]},{"osd":1,"uuid":"eeb3df13-66d8-4237-b42c-f41a4008730f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.32.75:6800","nonce":2396434981},{"type":"v1","addr":"10.249.32.75:6801","nonce":2396434981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.32.75:6802","nonce":2396434981},{"type":"v1","addr":"10.249.32.75:6803","nonce":2396434981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.32.75:6806","nonce":2396434981},{"type":"v1","addr":"10.249.32.75:6807","nonce":2396434981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.32.75:6804","nonce":2396434981},{"type":"v1","addr":"10.249.32.75:6805","nonce":2396434981}]},"public_addr":"10.249.32.75:6801/2396434981","cluster_addr":"10.249.32.75:6803/2396434981","heartbeat_back_addr":"10.249.32.75:6807/2396434981","heartbeat_front_addr":"10.249.32.75:6805/2396434981","state":["exists","up"]},{"osd":2,"uuid":"bf0df36b-48ee-4916-8267-de34c4981e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.32.76:6800","nonce":1980726265},{"type":"v1","addr":"10.249.32.76:6801","nonce":1980726265}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.32.76:6802","nonce":1980726265},{"type":"v1","addr":"10.249.32.76:6803","nonce":1980726265}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.32.76:6806","nonce":1980726265},{"type":"v1","addr":"10.249.32.76:6807","nonce":1980726265}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.32.76:6804","nonce":1980726265},{"type":"v1","addr":"10.249.32.76:6805","nonce":1980726265}]},"public_addr":"10.249.32.76:6801/1980726265","cluster_addr":"10.249.32.76:6803/1980726265","heartbeat_back_addr":"10.249.32.76:6807/1980726265","heartbeat_front_addr":"10.249.32.76:6805/1980726265","state":["exists","up"]},{"osd":3,"uuid":"109faadf-dfc8-44fb-b632-a56ca8b73443","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.34.80:6800","nonce":1532700234},{"type":"v1","addr":"10.249.34.80:6801","nonce":1532700234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.34.80:6802","nonce":1532700234},{"type":"v1","addr":"10.249.34.80:6803","nonce":1532700234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.34.80:6806","nonce":1532700234},{"type":"v1","addr":"10.249.34.80:6807","nonce":1532700234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.34.80:6804","nonce":1532700234},{"type":"v1","addr":"10.249.34.80:6805","nonce":1532700234}]},"public_addr":"10.249.34.80:6801/1532700234","cluster_addr":"10.249.34.80:6803/1532700234","heartbeat_back_addr":"10.249.34.80:6807/1532700234","heartbeat_front_addr":"10.249.34.80:6805/1532700234","state":["exists","up"]},{"osd":4,"uuid":"06149299-f29d-4f1a-b8ca-b4c274e41558","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.34.81:6800","nonce":3630779696},{"type":"v1","addr":"10.249.34.81:6801","nonce":3630779696}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.34.81:6802","nonce":3630779696},{"type":"v1","addr":"10.249.34.81:6803","nonce":3630779696}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.34.81:6806","nonce":3630779696},{"type":"v1","addr":"10.249.34.81:6807","nonce":3630779696}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.34.81:6804","nonce":3630779696},{"type":"v1","addr":"10.249.34.81:6805","nonce":3630779696}]},"public_addr":"10.249.34.81:6801/3630779696","cluster_addr":"10.249.34.81:6803/3630779696","heartbeat_back_addr":"10.249.34.81:6807/3630779696","heartbeat_front_addr":"10.249.34.81:6805/3630779696","state":["exists","up"]},{"osd":5,"uuid":"772999a7-ce8c-492e-b4a5-75e7ecd77b95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.34.82:6800","nonce":3725102274},{"type":"v1","addr":"10.249.34.82:6801","nonce":3725102274}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.34.82:6802","nonce":3725102274},{"type":"v1","addr":"10.249.34.82:6803","nonce":3725102274}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.34.82:6806","nonce":3725102274},{"type":"v1","addr":"10.249.34.82:6807","nonce":3725102274}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.34.82:6804","nonce":3725102274},{"type":"v1","addr":"10.249.34.82:6805","nonce":3725102274}]},"public_addr":"10.249.34.82:6801/3725102274","cluster_addr":"10.249.34.82:6803/3725102274","heartbeat_back_addr":"10.249.34.82:6807/3725102274","heartbeat_front_addr":"10.249.34.82:6805/3725102274","state":["exists","up"]},{"osd":6,"uuid":"42b0dfd3-0e2a-43fb-b166-5a7f0b57ade4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.36.140:6800","nonce":1429719811},{"type":"v1","addr":"10.249.36.140:6801","nonce":1429719811}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.36.140:6802","nonce":1429719811},{"type":"v1","addr":"10.249.36.140:6803","nonce":1429719811}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.36.140:6806","nonce":1429719811},{"type":"v1","addr":"10.249.36.140:6807","nonce":1429719811}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.36.140:6804","nonce":1429719811},{"type":"v1","addr":"10.249.36.140:6805","nonce":1429719811}]},"public_addr":"10.249.36.140:6801/1429719811","cluster_addr":"10.249.36.140:6803/1429719811","heartbeat_back_addr":"10.249.36.140:6807/1429719811","heartbeat_front_addr":"10.249.36.140:6805/1429719811","state":["exists","up"]},{"osd":7,"uuid":"aa7918fa-b0fe-4cce-86e0-b825160ccbab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.36.141:6800","nonce":3582135998},{"type":"v1","addr":"10.249.36.141:6801","nonce":3582135998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.36.141:6802","nonce":3582135998},{"type":"v1","addr":"10.249.36.141:6803","nonce":3582135998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.36.141:6806","nonce":3582135998},{"type":"v1","addr":"10.249.36.141:6807","nonce":3582135998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.36.141:6804","nonce":3582135998},{"type":"v1","addr":"10.249.36.141:6805","nonce":3582135998}]},"public_addr":"10.249.36.141:6801/3582135998","cluster_addr":"10.249.36.141:6803/3582135998","heartbeat_back_addr":"10.249.36.141:6807/3582135998","heartbeat_front_addr":"10.249.36.141:6805/3582135998","state":["exists","up"]},{"osd":8,"uuid":"8c3caec1-8e2b-4e29-b51e-7f0fb16b5804","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":29,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.32.78:6800","nonce":2902739700},{"type":"v1","addr":"10.249.32.78:6801","nonce":2902739700}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.32.78:6802","nonce":2902739700},{"type":"v1","addr":"10.249.32.78:6803","nonce":2902739700}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.32.78:6806","nonce":2902739700},{"type":"v1","addr":"10.249.32.78:6807","nonce":2902739700}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.32.78:6804","nonce":2902739700},{"type":"v1","addr":"10.249.32.78:6805","nonce":2902739700}]},"public_addr":"10.249.32.78:6801/2902739700","cluster_addr":"10.249.32.78:6803/2902739700","heartbeat_back_addr":"10.249.32.78:6807/2902739700","heartbeat_front_addr":"10.249.32.78:6805/2902739700","state":["exists","up"]},{"osd":9,"uuid":"7ec16300-d058-489d-b341-80c6f6375aa6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.36.143:6800","nonce":3569720756},{"type":"v1","addr":"10.249.36.143:6801","nonce":3569720756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.36.143:6802","nonce":3569720756},{"type":"v1","addr":"10.249.36.143:6803","nonce":3569720756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.36.143:6806","nonce":3569720756},{"type":"v1","addr":"10.249.36.143:6807","nonce":3569720756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.36.143:6804","nonce":3569720756},{"type":"v1","addr":"10.249.36.143:6805","nonce":3569720756}]},"public_addr":"10.249.36.143:6801/3569720756","cluster_addr":"10.249.36.143:6803/3569720756","heartbeat_back_addr":"10.249.36.143:6807/3569720756","heartbeat_front_addr":"10.249.36.143:6805/3569720756","state":["exists","up"]},{"osd":10,"uuid":"658b5090-fc93-4908-8889-f7b8ad1d5f46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.36.142:6800","nonce":2631742787},{"type":"v1","addr":"10.249.36.142:6801","nonce":2631742787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.36.142:6802","nonce":2631742787},{"type":"v1","addr":"10.249.36.142:6803","nonce":2631742787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.36.142:6806","nonce":2631742787},{"type":"v1","addr":"10.249.36.142:6807","nonce":2631742787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.36.142:6804","nonce":2631742787},{"type":"v1","addr":"10.249.36.142:6805","nonce":2631742787}]},"public_addr":"10.249.36.142:6801/2631742787","cluster_addr":"10.249.36.142:6803/2631742787","heartbeat_back_addr":"10.249.36.142:6807/2631742787","heartbeat_front_addr":"10.249.36.142:6805/2631742787","state":["exists","up"]},{"osd":11,"uuid":"9c3326a4-7ed4-49ed-937a-5c7a56125b33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":29,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.32.79:6800","nonce":2227166999},{"type":"v1","addr":"10.249.32.79:6801","nonce":2227166999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.32.79:6802","nonce":2227166999},{"type":"v1","addr":"10.249.32.79:6803","nonce":2227166999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.32.79:6806","nonce":2227166999},{"type":"v1","addr":"10.249.32.79:6807","nonce":2227166999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.32.79:6804","nonce":2227166999},{"type":"v1","addr":"10.249.32.79:6805","nonce":2227166999}]},"public_addr":"10.249.32.79:6801/2227166999","cluster_addr":"10.249.32.79:6803/2227166999","heartbeat_back_addr":"10.249.32.79:6807/2227166999","heartbeat_front_addr":"10.249.32.79:6805/2227166999","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-10T03:13:59.639154+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-10T03:14:08.444717+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-10T03:14:10.742204+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-10T03:14:03.219877+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-10T03:14:07.901081+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-10T03:14:09.584727+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-10T03:14:04.256752+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-10T03:14:10.209048+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-10T03:14:17.102473+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-10T03:14:10.182386+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-10T03:14:07.310478+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-10T03:14:17.200510+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.249.32.69:0/2108578815":"2022-03-11T03:12:22.565538+0000","10.249.32.69:6800/2959216379":"2022-03-11T03:12:22.565538+0000","10.249.32.69:6801/2959216379":"2022-03-11T03:12:22.565538+0000","10.249.32.69:0/1440386292":"2022-03-11T03:12:22.565538+0000","10.249.32.69:0/2380130538":"2022-03-11T03:12:22.565538+0000"},"erasure_code_profiles":{"default":{"k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-03-10T03:14:25.017 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph tell osd.0 flush_pg_stats 2022-03-10T03:14:25.018 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph tell osd.1 flush_pg_stats 2022-03-10T03:14:25.018 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph tell osd.2 flush_pg_stats 2022-03-10T03:14:25.019 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph tell osd.3 flush_pg_stats 2022-03-10T03:14:25.020 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph tell osd.4 flush_pg_stats 2022-03-10T03:14:25.020 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph tell osd.5 flush_pg_stats 2022-03-10T03:14:25.021 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph tell osd.6 flush_pg_stats 2022-03-10T03:14:25.021 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph tell osd.7 flush_pg_stats 2022-03-10T03:14:25.022 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph tell osd.8 flush_pg_stats 2022-03-10T03:14:25.023 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph tell osd.9 flush_pg_stats 2022-03-10T03:14:25.023 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph tell osd.10 flush_pg_stats 2022-03-10T03:14:25.024 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph tell osd.11 flush_pg_stats 2022-03-10T03:14:25.611 INFO:teuthology.orchestra.run.smithi037.stdout:73014444039 2022-03-10T03:14:25.612 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph osd last-stat-seq osd.0 2022-03-10T03:14:25.699 INFO:teuthology.orchestra.run.smithi037.stdout:103079215109 2022-03-10T03:14:25.700 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph osd last-stat-seq osd.9 2022-03-10T03:14:25.896 INFO:teuthology.orchestra.run.smithi037.stdout:103079215111 2022-03-10T03:14:25.897 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph osd last-stat-seq osd.1 2022-03-10T03:14:26.060 INFO:teuthology.orchestra.run.smithi037.stdout:94489280517 2022-03-10T03:14:26.061 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph osd last-stat-seq osd.10 2022-03-10T03:14:26.120 INFO:teuthology.orchestra.run.smithi037.stdout:103079215109 2022-03-10T03:14:26.121 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph osd last-stat-seq osd.7 2022-03-10T03:14:26.129 INFO:teuthology.orchestra.run.smithi037.stdout:98784247814 2022-03-10T03:14:26.130 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph osd last-stat-seq osd.4 2022-03-10T03:14:26.131 INFO:teuthology.orchestra.run.smithi037.stdout:90194313223 2022-03-10T03:14:26.132 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph osd last-stat-seq osd.6 2022-03-10T03:14:26.134 INFO:teuthology.orchestra.run.smithi037.stdout:107374182404 2022-03-10T03:14:26.134 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph osd last-stat-seq osd.2 2022-03-10T03:14:26.138 INFO:teuthology.orchestra.run.smithi037.stdout:103079215109 2022-03-10T03:14:26.138 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph osd last-stat-seq osd.5 2022-03-10T03:14:26.189 INFO:teuthology.orchestra.run.smithi037.stdout:85899345927 2022-03-10T03:14:26.190 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph osd last-stat-seq osd.3 2022-03-10T03:14:26.253 INFO:teuthology.orchestra.run.smithi037.stdout:124554051588 2022-03-10T03:14:26.253 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph osd last-stat-seq osd.11 2022-03-10T03:14:26.398 INFO:teuthology.orchestra.run.smithi037.stdout:73014444038 2022-03-10T03:14:26.435 INFO:tasks.rook.ceph_manager.ceph:need seq 73014444039 got 73014444038 for osd.0 2022-03-10T03:14:26.626 INFO:teuthology.orchestra.run.smithi037.stdout:103079215108 2022-03-10T03:14:26.684 INFO:teuthology.orchestra.run.smithi037.stdout:103079215110 2022-03-10T03:14:26.783 INFO:teuthology.orchestra.run.smithi037.stdout:94489280516 2022-03-10T03:14:26.899 INFO:tasks.rook.ceph_manager.ceph:need seq 103079215109 got 103079215108 for osd.9 2022-03-10T03:14:26.909 INFO:tasks.rook.ceph_manager.ceph:need seq 103079215111 got 103079215110 for osd.1 2022-03-10T03:14:26.913 INFO:tasks.rook.ceph_manager.ceph:need seq 94489280517 got 94489280516 for osd.10 2022-03-10T03:14:26.918 INFO:teuthology.orchestra.run.smithi037.stdout:124554051589 2022-03-10T03:14:26.919 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph osd last-stat-seq osd.8 2022-03-10T03:14:26.970 INFO:teuthology.orchestra.run.smithi037.stdout:103079215108 2022-03-10T03:14:27.000 INFO:teuthology.orchestra.run.smithi037.stdout:90194313222 2022-03-10T03:14:27.032 INFO:tasks.rook.ceph_manager.ceph:need seq 103079215109 got 103079215108 for osd.7 2022-03-10T03:14:27.044 INFO:tasks.rook.ceph_manager.ceph:need seq 90194313223 got 90194313222 for osd.6 2022-03-10T03:14:27.149 INFO:teuthology.orchestra.run.smithi037.stdout:98784247813 2022-03-10T03:14:27.183 INFO:tasks.rook.ceph_manager.ceph:need seq 98784247814 got 98784247813 for osd.4 2022-03-10T03:14:27.194 INFO:teuthology.orchestra.run.smithi037.stdout:107374182402 2022-03-10T03:14:27.214 INFO:tasks.rook.ceph_manager.ceph:need seq 107374182404 got 107374182402 for osd.2 2022-03-10T03:14:27.235 INFO:teuthology.orchestra.run.smithi037.stdout:85899345926 2022-03-10T03:14:27.263 INFO:tasks.rook.ceph_manager.ceph:need seq 85899345927 got 85899345926 for osd.3 2022-03-10T03:14:27.304 INFO:teuthology.orchestra.run.smithi037.stdout:124554051587 2022-03-10T03:14:27.328 INFO:teuthology.orchestra.run.smithi037.stdout:103079215108 2022-03-10T03:14:27.330 INFO:tasks.rook.ceph_manager.ceph:need seq 124554051588 got 124554051587 for osd.11 2022-03-10T03:14:27.352 INFO:tasks.rook.ceph_manager.ceph:need seq 103079215109 got 103079215108 for osd.5 2022-03-10T03:14:27.437 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph osd last-stat-seq osd.0 2022-03-10T03:14:27.510 INFO:teuthology.orchestra.run.smithi037.stdout:124554051588 2022-03-10T03:14:27.533 INFO:tasks.rook.ceph_manager.ceph:need seq 124554051589 got 124554051588 for osd.8 2022-03-10T03:14:27.900 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph osd last-stat-seq osd.9 2022-03-10T03:14:27.910 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph osd last-stat-seq osd.1 2022-03-10T03:14:27.914 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph osd last-stat-seq osd.10 2022-03-10T03:14:27.976 INFO:teuthology.orchestra.run.smithi037.stdout:73014444039 2022-03-10T03:14:27.996 INFO:tasks.rook.ceph_manager.ceph:need seq 73014444039 got 73014444039 for osd.0 2022-03-10T03:14:27.996 DEBUG:teuthology.parallel:result is None 2022-03-10T03:14:28.033 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph osd last-stat-seq osd.7 2022-03-10T03:14:28.045 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph osd last-stat-seq osd.6 2022-03-10T03:14:28.184 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph osd last-stat-seq osd.4 2022-03-10T03:14:28.215 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph osd last-stat-seq osd.2 2022-03-10T03:14:28.263 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph osd last-stat-seq osd.3 2022-03-10T03:14:28.331 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph osd last-stat-seq osd.11 2022-03-10T03:14:28.353 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph osd last-stat-seq osd.5 2022-03-10T03:14:28.535 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph osd last-stat-seq osd.8 2022-03-10T03:14:28.550 INFO:teuthology.orchestra.run.smithi037.stdout:103079215109 2022-03-10T03:14:28.736 INFO:tasks.rook.ceph_manager.ceph:need seq 103079215109 got 103079215109 for osd.9 2022-03-10T03:14:28.737 DEBUG:teuthology.parallel:result is None 2022-03-10T03:14:28.746 INFO:teuthology.orchestra.run.smithi037.stdout:103079215111 2022-03-10T03:14:28.794 INFO:teuthology.orchestra.run.smithi037.stdout:94489280518 2022-03-10T03:14:28.864 INFO:tasks.rook.ceph_manager.ceph:need seq 103079215111 got 103079215111 for osd.1 2022-03-10T03:14:28.865 DEBUG:teuthology.parallel:result is None 2022-03-10T03:14:28.868 INFO:tasks.rook.ceph_manager.ceph:need seq 94489280517 got 94489280518 for osd.10 2022-03-10T03:14:28.868 DEBUG:teuthology.parallel:result is None 2022-03-10T03:14:28.889 INFO:teuthology.orchestra.run.smithi037.stdout:103079215109 2022-03-10T03:14:28.915 INFO:tasks.rook.ceph_manager.ceph:need seq 103079215109 got 103079215109 for osd.7 2022-03-10T03:14:28.916 DEBUG:teuthology.parallel:result is None 2022-03-10T03:14:28.939 INFO:teuthology.orchestra.run.smithi037.stdout:90194313223 2022-03-10T03:14:28.969 INFO:tasks.rook.ceph_manager.ceph:need seq 90194313223 got 90194313223 for osd.6 2022-03-10T03:14:28.970 DEBUG:teuthology.parallel:result is None 2022-03-10T03:14:29.072 INFO:teuthology.orchestra.run.smithi037.stdout:98784247814 2022-03-10T03:14:29.093 INFO:tasks.rook.ceph_manager.ceph:need seq 98784247814 got 98784247814 for osd.4 2022-03-10T03:14:29.093 DEBUG:teuthology.parallel:result is None 2022-03-10T03:14:29.162 INFO:teuthology.orchestra.run.smithi037.stdout:107374182404 2022-03-10T03:14:29.165 INFO:teuthology.orchestra.run.smithi037.stdout:85899345928 2022-03-10T03:14:29.185 INFO:tasks.rook.ceph_manager.ceph:need seq 107374182404 got 107374182404 for osd.2 2022-03-10T03:14:29.185 DEBUG:teuthology.parallel:result is None 2022-03-10T03:14:29.187 INFO:tasks.rook.ceph_manager.ceph:need seq 85899345927 got 85899345928 for osd.3 2022-03-10T03:14:29.187 DEBUG:teuthology.parallel:result is None 2022-03-10T03:14:29.257 INFO:teuthology.orchestra.run.smithi037.stdout:103079215110 2022-03-10T03:14:29.282 INFO:tasks.rook.ceph_manager.ceph:need seq 103079215109 got 103079215110 for osd.5 2022-03-10T03:14:29.282 DEBUG:teuthology.parallel:result is None 2022-03-10T03:14:29.285 INFO:teuthology.orchestra.run.smithi037.stdout:124554051589 2022-03-10T03:14:29.306 INFO:tasks.rook.ceph_manager.ceph:need seq 124554051589 got 124554051589 for osd.8 2022-03-10T03:14:29.306 DEBUG:teuthology.parallel:result is None 2022-03-10T03:14:29.372 INFO:teuthology.orchestra.run.smithi037.stdout:124554051588 2022-03-10T03:14:29.393 INFO:tasks.rook.ceph_manager.ceph:need seq 124554051588 got 124554051588 for osd.11 2022-03-10T03:14:29.394 DEBUG:teuthology.parallel:result is None 2022-03-10T03:14:29.394 INFO:tasks.rook.ceph_manager.ceph:waiting for clean 2022-03-10T03:14:29.394 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph pg dump --format=json 2022-03-10T03:14:29.908 INFO:teuthology.orchestra.run.smithi037.stdout: 2022-03-10T03:14:29.910 INFO:teuthology.orchestra.run.smithi037.stderr:dumped all 2022-03-10T03:14:29.928 INFO:teuthology.orchestra.run.smithi037.stdout:{"pg_ready":true,"pg_map":{"version":84,"stamp":"2022-03-10T03:14:28.970080+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":167,"num_write_kb":2676,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":125,"ondisk_log_size":125,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":12,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":1124941824,"kb_used":65848,"kb_used_data":4784,"kb_used_omap":0,"kb_used_meta":59840,"kb_avail":1124875976,"statfs":{"total":1151940427776,"available":1151872999424,"internally_reserved":0,"allocated":4898816,"data_stored":2832421,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":61276160},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":20,"num_read_kb":18,"num_write":44,"num_write_kb":1044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003064"},"pg_stats":[{"pgid":"1.0","version":"30'125","reported_seq":173,"reported_epoch":30,"state":"active+clean","last_fresh":"2022-03-10T03:14:24.444500+0000","last_change":"2022-03-10T03:14:17.274139+0000","last_active":"2022-03-10T03:14:24.444500+0000","last_peered":"2022-03-10T03:14:24.444500+0000","last_clean":"2022-03-10T03:14:24.444500+0000","last_became_active":"2022-03-10T03:14:17.271999+0000","last_became_peered":"2022-03-10T03:14:17.271999+0000","last_unstale":"2022-03-10T03:14:24.444500+0000","last_undegraded":"2022-03-10T03:14:24.444500+0000","last_fullsized":"2022-03-10T03:14:24.444500+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-10T03:14:09.814348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-10T03:14:09.814348+0000","last_clean_scrub_stamp":"2022-03-10T03:14:09.814348+0000","objects_scrubbed":0,"log_size":125,"ondisk_log_size":125,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-11T03:23:23.860931+0000","stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":167,"num_write_kb":2676,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,10],"acting":[7,2,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":167,"num_write_kb":2676,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1413120,"data_stored":1402464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":125,"ondisk_log_size":125,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":11,"up_from":29,"seq":124554051588,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5092,"kb_used_data":284,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":93740060,"statfs":{"total":95995035648,"available":95989821440,"internally_reserved":0,"allocated":290816,"data_stored":119234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.103}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4370000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1459999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1839999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5980000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8220000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0259999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.391}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.98}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.363}]}]},{"osd":8,"up_from":29,"seq":124554051589,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5092,"kb_used_data":284,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":93740060,"statfs":{"total":95995035648,"available":95989821440,"internally_reserved":0,"allocated":290816,"data_stored":119234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.8520000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.468}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9730000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.234}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.4609999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.677}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.0049999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.3679999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1309999999999998}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.1509999999999998}]}]},{"osd":2,"up_from":25,"seq":107374182404,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6504,"kb_used_data":744,"kb_used_omap":0,"kb_used_meta":5376,"kb_avail":93738648,"statfs":{"total":95995035648,"available":95988375552,"internally_reserved":0,"allocated":761856,"data_stored":586722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5505024},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0900000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.216}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.468}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6279999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.524}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9570000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.978}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.907}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1520000000000001}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2}]}]},{"osd":9,"up_from":24,"seq":103079215110,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5156,"kb_used_data":284,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93739996,"statfs":{"total":95995035648,"available":95989755904,"internally_reserved":0,"allocated":290816,"data_stored":119234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.857}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0859999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.145}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2610000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.0699999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.7090000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.5219999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.2509999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.1600000000000001}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.524}]}]},{"osd":7,"up_from":24,"seq":103079215110,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6504,"kb_used_data":744,"kb_used_omap":0,"kb_used_meta":5376,"kb_avail":93738648,"statfs":{"total":95995035648,"available":95988375552,"internally_reserved":0,"allocated":761856,"data_stored":586722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5505024},"hb_peers":[0,1,2,3,4,5,6,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0329999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.196}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5009999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.698}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.591}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.7530000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4099999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8260000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1869999999999998}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8979999999999999}]}]},{"osd":5,"up_from":24,"seq":103079215110,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5156,"kb_used_data":284,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93739996,"statfs":{"total":95995035648,"available":95989755904,"internally_reserved":0,"allocated":290816,"data_stored":119234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.3410000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5029999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.589}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.742}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.7120000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4319999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.393}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6539999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.6549999999999998}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6220000000000001}]}]},{"osd":3,"up_from":20,"seq":85899345928,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5156,"kb_used_data":284,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93739996,"statfs":{"total":95995035648,"available":95989755904,"internally_reserved":0,"allocated":290816,"data_stored":119234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.9779999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.194}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.9189999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.6509999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.856999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.827}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.8659999999999997}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.8920000000000003}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.4640000000000004}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.784000000000001}]}]},{"osd":10,"up_from":22,"seq":94489280518,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6504,"kb_used_data":744,"kb_used_omap":0,"kb_used_meta":5376,"kb_avail":93738648,"statfs":{"total":95995035648,"available":95988375552,"internally_reserved":0,"allocated":761856,"data_stored":586722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5505024},"hb_peers":[0,1,2,3,4,5,6,7,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63900000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4339999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.298}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92200000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.016}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.534}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.516}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.048}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5629999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.607}]}]},{"osd":0,"up_from":17,"seq":73014444040,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5220,"kb_used_data":284,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739932,"statfs":{"total":95995035648,"available":95989690368,"internally_reserved":0,"allocated":290816,"data_stored":119234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4339999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5880000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.393}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.089}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8380000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2829999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0579999999999998}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.383}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.9510000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2109999999999999}]}]},{"osd":6,"up_from":21,"seq":90194313223,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5152,"kb_used_data":280,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93740000,"statfs":{"total":95995035648,"available":95989760000,"internally_reserved":0,"allocated":286720,"data_stored":118383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94399999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98099999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.196}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4370000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.5049999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.756}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5529999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8620000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7350000000000001}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9199999999999999}]}]},{"osd":4,"up_from":23,"seq":98784247815,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5156,"kb_used_data":284,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93739996,"statfs":{"total":95995035648,"available":95989755904,"internally_reserved":0,"allocated":290816,"data_stored":119234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.075}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8080000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5449999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.704}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3780000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.27}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.327}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0230000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.403}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2480000000000002}]}]},{"osd":1,"up_from":24,"seq":103079215112,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5156,"kb_used_data":284,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93739996,"statfs":{"total":95995035648,"available":95989755904,"internally_reserved":0,"allocated":290816,"data_stored":119234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76600000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.125}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0920000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4199999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.4340000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.645}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5189999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8380000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9039999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7609999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":471040,"data_stored":467488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":471040,"data_stored":467488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":471040,"data_stored":467488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2022-03-10T03:14:29.930 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph pg dump --format=json 2022-03-10T03:14:30.468 INFO:teuthology.orchestra.run.smithi037.stdout: 2022-03-10T03:14:30.471 INFO:teuthology.orchestra.run.smithi037.stderr:dumped all 2022-03-10T03:14:30.488 INFO:teuthology.orchestra.run.smithi037.stdout:{"pg_ready":true,"pg_map":{"version":84,"stamp":"2022-03-10T03:14:28.970080+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":167,"num_write_kb":2676,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":125,"ondisk_log_size":125,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":12,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":1124941824,"kb_used":65848,"kb_used_data":4784,"kb_used_omap":0,"kb_used_meta":59840,"kb_avail":1124875976,"statfs":{"total":1151940427776,"available":1151872999424,"internally_reserved":0,"allocated":4898816,"data_stored":2832421,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":61276160},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":20,"num_read_kb":18,"num_write":44,"num_write_kb":1044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003064"},"pg_stats":[{"pgid":"1.0","version":"30'125","reported_seq":173,"reported_epoch":30,"state":"active+clean","last_fresh":"2022-03-10T03:14:24.444500+0000","last_change":"2022-03-10T03:14:17.274139+0000","last_active":"2022-03-10T03:14:24.444500+0000","last_peered":"2022-03-10T03:14:24.444500+0000","last_clean":"2022-03-10T03:14:24.444500+0000","last_became_active":"2022-03-10T03:14:17.271999+0000","last_became_peered":"2022-03-10T03:14:17.271999+0000","last_unstale":"2022-03-10T03:14:24.444500+0000","last_undegraded":"2022-03-10T03:14:24.444500+0000","last_fullsized":"2022-03-10T03:14:24.444500+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-10T03:14:09.814348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-10T03:14:09.814348+0000","last_clean_scrub_stamp":"2022-03-10T03:14:09.814348+0000","objects_scrubbed":0,"log_size":125,"ondisk_log_size":125,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-11T03:23:23.860931+0000","stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":167,"num_write_kb":2676,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,10],"acting":[7,2,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":167,"num_write_kb":2676,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1413120,"data_stored":1402464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":125,"ondisk_log_size":125,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":11,"up_from":29,"seq":124554051588,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5092,"kb_used_data":284,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":93740060,"statfs":{"total":95995035648,"available":95989821440,"internally_reserved":0,"allocated":290816,"data_stored":119234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.103}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4370000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1459999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1839999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5980000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8220000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0259999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.391}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.98}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.363}]}]},{"osd":8,"up_from":29,"seq":124554051589,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5092,"kb_used_data":284,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":93740060,"statfs":{"total":95995035648,"available":95989821440,"internally_reserved":0,"allocated":290816,"data_stored":119234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.8520000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.468}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9730000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.234}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.4609999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.677}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.0049999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.3679999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1309999999999998}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.1509999999999998}]}]},{"osd":2,"up_from":25,"seq":107374182404,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6504,"kb_used_data":744,"kb_used_omap":0,"kb_used_meta":5376,"kb_avail":93738648,"statfs":{"total":95995035648,"available":95988375552,"internally_reserved":0,"allocated":761856,"data_stored":586722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5505024},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0900000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.216}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.468}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6279999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.524}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9570000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.978}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.907}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1520000000000001}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2}]}]},{"osd":9,"up_from":24,"seq":103079215110,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5156,"kb_used_data":284,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93739996,"statfs":{"total":95995035648,"available":95989755904,"internally_reserved":0,"allocated":290816,"data_stored":119234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.857}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0859999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.145}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2610000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.0699999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.7090000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.5219999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.2509999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.1600000000000001}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.524}]}]},{"osd":7,"up_from":24,"seq":103079215110,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6504,"kb_used_data":744,"kb_used_omap":0,"kb_used_meta":5376,"kb_avail":93738648,"statfs":{"total":95995035648,"available":95988375552,"internally_reserved":0,"allocated":761856,"data_stored":586722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5505024},"hb_peers":[0,1,2,3,4,5,6,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0329999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.196}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5009999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.698}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.591}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.7530000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4099999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8260000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1869999999999998}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8979999999999999}]}]},{"osd":5,"up_from":24,"seq":103079215110,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5156,"kb_used_data":284,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93739996,"statfs":{"total":95995035648,"available":95989755904,"internally_reserved":0,"allocated":290816,"data_stored":119234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.3410000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5029999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.589}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.742}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.7120000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4319999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.393}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6539999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.6549999999999998}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6220000000000001}]}]},{"osd":3,"up_from":20,"seq":85899345928,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5156,"kb_used_data":284,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93739996,"statfs":{"total":95995035648,"available":95989755904,"internally_reserved":0,"allocated":290816,"data_stored":119234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.9779999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.194}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.9189999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.6509999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.856999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.827}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.8659999999999997}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.8920000000000003}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.4640000000000004}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.784000000000001}]}]},{"osd":10,"up_from":22,"seq":94489280518,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6504,"kb_used_data":744,"kb_used_omap":0,"kb_used_meta":5376,"kb_avail":93738648,"statfs":{"total":95995035648,"available":95988375552,"internally_reserved":0,"allocated":761856,"data_stored":586722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5505024},"hb_peers":[0,1,2,3,4,5,6,7,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63900000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4339999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.298}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92200000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.016}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.534}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.516}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.048}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5629999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.607}]}]},{"osd":0,"up_from":17,"seq":73014444040,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5220,"kb_used_data":284,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739932,"statfs":{"total":95995035648,"available":95989690368,"internally_reserved":0,"allocated":290816,"data_stored":119234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4339999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5880000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.393}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.089}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8380000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2829999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0579999999999998}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.383}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.9510000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2109999999999999}]}]},{"osd":6,"up_from":21,"seq":90194313223,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5152,"kb_used_data":280,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93740000,"statfs":{"total":95995035648,"available":95989760000,"internally_reserved":0,"allocated":286720,"data_stored":118383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94399999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98099999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.196}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4370000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.5049999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.756}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5529999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8620000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7350000000000001}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9199999999999999}]}]},{"osd":4,"up_from":23,"seq":98784247815,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5156,"kb_used_data":284,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93739996,"statfs":{"total":95995035648,"available":95989755904,"internally_reserved":0,"allocated":290816,"data_stored":119234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.075}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8080000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5449999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.704}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3780000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.27}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.327}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0230000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.403}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2480000000000002}]}]},{"osd":1,"up_from":24,"seq":103079215112,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5156,"kb_used_data":284,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93739996,"statfs":{"total":95995035648,"available":95989755904,"internally_reserved":0,"allocated":290816,"data_stored":119234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76600000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.125}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0920000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4199999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.4340000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.645}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5189999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8380000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9039999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7609999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":471040,"data_stored":467488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":471040,"data_stored":467488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":471040,"data_stored":467488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2022-03-10T03:14:30.490 INFO:tasks.rook.ceph_manager.ceph:clean! 2022-03-10T03:14:30.491 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2022-03-10T03:14:30.491 INFO:tasks.rook.ceph_manager.ceph:wait_until_healthy 2022-03-10T03:14:30.491 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph health --format=json 2022-03-10T03:14:31.093 INFO:teuthology.orchestra.run.smithi037.stdout: 2022-03-10T03:14:31.094 INFO:teuthology.orchestra.run.smithi037.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2022-03-10T03:14:31.146 INFO:tasks.rook.ceph_manager.ceph:wait_until_healthy done 2022-03-10T03:14:31.147 INFO:tasks.rook:Rook complete, yielding 2022-03-10T03:14:31.147 INFO:teuthology.run_tasks:Running task rook.shell... 2022-03-10T03:14:31.158 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph -s 2022-03-10T03:14:31.772 INFO:teuthology.orchestra.run.smithi037.stdout: cluster: 2022-03-10T03:14:31.772 INFO:teuthology.orchestra.run.smithi037.stdout: id: 2d9bcb02-6465-4ca2-9864-d3ce67476d09 2022-03-10T03:14:31.773 INFO:teuthology.orchestra.run.smithi037.stdout: health: HEALTH_OK 2022-03-10T03:14:31.773 INFO:teuthology.orchestra.run.smithi037.stdout: 2022-03-10T03:14:31.773 INFO:teuthology.orchestra.run.smithi037.stdout: services: 2022-03-10T03:14:31.773 INFO:teuthology.orchestra.run.smithi037.stdout: mon: 3 daemons, quorum a,b,c (age 2m) 2022-03-10T03:14:31.773 INFO:teuthology.orchestra.run.smithi037.stdout: mgr: a(active, since 2m) 2022-03-10T03:14:31.774 INFO:teuthology.orchestra.run.smithi037.stdout: osd: 12 osds: 12 up (since 13s), 12 in (since 60s) 2022-03-10T03:14:31.774 INFO:teuthology.orchestra.run.smithi037.stdout: 2022-03-10T03:14:31.774 INFO:teuthology.orchestra.run.smithi037.stdout: data: 2022-03-10T03:14:31.774 INFO:teuthology.orchestra.run.smithi037.stdout: pools: 1 pools, 1 pgs 2022-03-10T03:14:31.775 INFO:teuthology.orchestra.run.smithi037.stdout: objects: 2 objects, 577 KiB 2022-03-10T03:14:31.775 INFO:teuthology.orchestra.run.smithi037.stdout: usage: 64 MiB used, 1.0 TiB / 1.0 TiB avail 2022-03-10T03:14:31.775 INFO:teuthology.orchestra.run.smithi037.stdout: pgs: 1 active+clean 2022-03-10T03:14:31.775 INFO:teuthology.orchestra.run.smithi037.stdout: 2022-03-10T03:14:31.775 INFO:teuthology.orchestra.run.smithi037.stdout: io: 2022-03-10T03:14:31.776 INFO:teuthology.orchestra.run.smithi037.stdout: client: 1.7 KiB/s rd, 97 KiB/s wr, 1 op/s rd, 4 op/s wr 2022-03-10T03:14:31.776 INFO:teuthology.orchestra.run.smithi037.stdout: 2022-03-10T03:14:31.792 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph orch status 2022-03-10T03:14:32.516 INFO:teuthology.orchestra.run.smithi037.stdout:Backend: rook 2022-03-10T03:14:32.517 INFO:teuthology.orchestra.run.smithi037.stdout:Available: Yes 2022-03-10T03:14:32.545 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph orch ps 2022-03-10T03:14:33.149 INFO:teuthology.orchestra.run.smithi037.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID 2022-03-10T03:14:33.150 INFO:teuthology.orchestra.run.smithi037.stdout:crashcollector.crash smithi037 running (62s) 0s ago 62s - - 4b871cde5e1a 2022-03-10T03:14:33.150 INFO:teuthology.orchestra.run.smithi037.stdout:crashcollector.crash smithi038 running (2m) 0s ago 2m - - 4b871cde5e1a 2022-03-10T03:14:33.150 INFO:teuthology.orchestra.run.smithi037.stdout:crashcollector.crash smithi195 running (2m) 0s ago 2m - - 4b871cde5e1a 2022-03-10T03:14:33.151 INFO:teuthology.orchestra.run.smithi037.stdout:mgr.a smithi037 running (2m) 0s ago 2m - - 4b871cde5e1a 2022-03-10T03:14:33.151 INFO:teuthology.orchestra.run.smithi037.stdout:mon.a smithi038 running (5m) 0s ago 5m - - 4b871cde5e1a 2022-03-10T03:14:33.151 INFO:teuthology.orchestra.run.smithi037.stdout:mon.b smithi195 running (3m) 0s ago 3m - - 4b871cde5e1a 2022-03-10T03:14:33.151 INFO:teuthology.orchestra.run.smithi037.stdout:mon.c smithi037 running (3m) 0s ago 3m - - 4b871cde5e1a 2022-03-10T03:14:33.151 INFO:teuthology.orchestra.run.smithi037.stdout:osd.0 smithi038 running (67s) 0s ago 67s - - 4b871cde5e1a 2022-03-10T03:14:33.152 INFO:teuthology.orchestra.run.smithi037.stdout:osd.1 smithi037 running (62s) 0s ago 63s - - 4b871cde5e1a 2022-03-10T03:14:33.152 INFO:teuthology.orchestra.run.smithi037.stdout:osd.10 smithi195 running (53s) 0s ago 53s - - 4b871cde5e1a 2022-03-10T03:14:33.152 INFO:teuthology.orchestra.run.smithi037.stdout:osd.11 smithi037 running (49s) 0s ago 49s - - 4b871cde5e1a 2022-03-10T03:14:33.152 INFO:teuthology.orchestra.run.smithi037.stdout:osd.2 smithi037 running (61s) 0s ago 62s - - 4b871cde5e1a 2022-03-10T03:14:33.153 INFO:teuthology.orchestra.run.smithi037.stdout:osd.3 smithi038 running (59s) 0s ago 59s - - 4b871cde5e1a 2022-03-10T03:14:33.153 INFO:teuthology.orchestra.run.smithi037.stdout:osd.4 smithi038 running (57s) 0s ago 57s - - 4b871cde5e1a 2022-03-10T03:14:33.153 INFO:teuthology.orchestra.run.smithi037.stdout:osd.5 smithi038 running (54s) 0s ago 55s - - 4b871cde5e1a 2022-03-10T03:14:33.153 INFO:teuthology.orchestra.run.smithi037.stdout:osd.6 smithi195 running (58s) 0s ago 58s - - 4b871cde5e1a 2022-03-10T03:14:33.153 INFO:teuthology.orchestra.run.smithi037.stdout:osd.7 smithi195 running (54s) 0s ago 54s - - 4b871cde5e1a 2022-03-10T03:14:33.154 INFO:teuthology.orchestra.run.smithi037.stdout:osd.8 smithi037 running (50s) 0s ago 50s - - 4b871cde5e1a 2022-03-10T03:14:33.154 INFO:teuthology.orchestra.run.smithi037.stdout:osd.9 smithi195 running (51s) 0s ago 52s - - 4b871cde5e1a 2022-03-10T03:14:33.169 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph orch ls 2022-03-10T03:14:33.929 INFO:teuthology.orchestra.run.smithi037.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2022-03-10T03:14:33.930 INFO:teuthology.orchestra.run.smithi037.stdout:crash 3/3 0s ago 2m * 2022-03-10T03:14:33.930 INFO:teuthology.orchestra.run.smithi037.stdout:mgr 1/1 0s ago 2m count:1 2022-03-10T03:14:33.930 INFO:teuthology.orchestra.run.smithi037.stdout:mon 3/3 0s ago 5m count:3 2022-03-10T03:14:33.931 INFO:teuthology.orchestra.run.smithi037.stdout:osd 24 0s ago 67s 2022-03-10T03:14:33.931 INFO:teuthology.orchestra.run.smithi037.stdout:osd.all-available-devices 0 0s ago - * 2022-03-10T03:14:33.956 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph orch device ls 2022-03-10T03:14:34.501 INFO:teuthology.orchestra.run.smithi037.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2022-03-10T03:14:34.501 INFO:teuthology.orchestra.run.smithi037.stdout:smithi037 /dev/vg_nvme/lv_1 unknown None 107G 0s ago 2022-03-10T03:14:34.501 INFO:teuthology.orchestra.run.smithi037.stdout:smithi037 /dev/vg_nvme/lv_2 unknown None 107G 0s ago 2022-03-10T03:14:34.501 INFO:teuthology.orchestra.run.smithi037.stdout:smithi037 /dev/vg_nvme/lv_3 unknown None 107G 0s ago 2022-03-10T03:14:34.502 INFO:teuthology.orchestra.run.smithi037.stdout:smithi037 /dev/vg_nvme/lv_4 unknown None 107G 0s ago 2022-03-10T03:14:34.502 INFO:teuthology.orchestra.run.smithi037.stdout:smithi038 /dev/vg_nvme/lv_1 unknown None 107G 0s ago 2022-03-10T03:14:34.502 INFO:teuthology.orchestra.run.smithi037.stdout:smithi038 /dev/vg_nvme/lv_2 unknown None 107G 0s ago 2022-03-10T03:14:34.502 INFO:teuthology.orchestra.run.smithi037.stdout:smithi038 /dev/vg_nvme/lv_3 unknown None 107G 0s ago 2022-03-10T03:14:34.503 INFO:teuthology.orchestra.run.smithi037.stdout:smithi038 /dev/vg_nvme/lv_4 unknown None 107G 0s ago 2022-03-10T03:14:34.503 INFO:teuthology.orchestra.run.smithi037.stdout:smithi195 /dev/vg_nvme/lv_1 unknown None 107G 0s ago 2022-03-10T03:14:34.503 INFO:teuthology.orchestra.run.smithi037.stdout:smithi195 /dev/vg_nvme/lv_2 unknown None 107G 0s ago 2022-03-10T03:14:34.503 INFO:teuthology.orchestra.run.smithi037.stdout:smithi195 /dev/vg_nvme/lv_3 unknown None 107G 0s ago 2022-03-10T03:14:34.503 INFO:teuthology.orchestra.run.smithi037.stdout:smithi195 /dev/vg_nvme/lv_4 unknown None 107G 0s ago 2022-03-10T03:14:34.525 INFO:teuthology.run_tasks:Running task exec... 2022-03-10T03:14:34.538 INFO:teuthology.task.exec:Executing custom commands... 2022-03-10T03:14:34.539 INFO:teuthology.task.exec:Running commands on role host.a host ubuntu@smithi037.front.sepia.ceph.com 2022-03-10T03:14:34.539 DEBUG:teuthology.orchestra.run.smithi037:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'set -ex 2022-03-10T03:14:34.539 DEBUG:teuthology.orchestra.run.smithi037:> toolbox() { 2022-03-10T03:14:34.540 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- "$@" 2022-03-10T03:14:34.540 DEBUG:teuthology.orchestra.run.smithi037:> } 2022-03-10T03:14:34.540 DEBUG:teuthology.orchestra.run.smithi037:> orig_num_osd=`toolbox ceph osd stat | cut -f3 -d " "` 2022-03-10T03:14:34.541 DEBUG:teuthology.orchestra.run.smithi037:> toolbox ceph orch osd rm 0 --force 2022-03-10T03:14:34.541 DEBUG:teuthology.orchestra.run.smithi037:> removed_pv="" 2022-03-10T03:14:34.541 DEBUG:teuthology.orchestra.run.smithi037:> while [ "$removed_pv" = "" ] 2022-03-10T03:14:34.541 DEBUG:teuthology.orchestra.run.smithi037:> do 2022-03-10T03:14:34.542 DEBUG:teuthology.orchestra.run.smithi037:> removed_pv=`kubectl get pv | grep Released | cut -f1 -d " "` 2022-03-10T03:14:34.542 DEBUG:teuthology.orchestra.run.smithi037:> sleep 3s 2022-03-10T03:14:34.542 DEBUG:teuthology.orchestra.run.smithi037:> done 2022-03-10T03:14:34.543 DEBUG:teuthology.orchestra.run.smithi037:> target_path=`kubectl get pv $removed_pv -o jsonpath='"'"'{.spec.local.path}'"'"'` 2022-03-10T03:14:34.543 DEBUG:teuthology.orchestra.run.smithi037:> host=`echo $removed_pv | cut -f1 -d "-"` 2022-03-10T03:14:34.543 DEBUG:teuthology.orchestra.run.smithi037:> toolbox ceph orch device zap $host $target_path --force 2022-03-10T03:14:34.544 DEBUG:teuthology.orchestra.run.smithi037:> zap_completion="0" 2022-03-10T03:14:34.544 DEBUG:teuthology.orchestra.run.smithi037:> while [ "$zap_completion" = "0" ] 2022-03-10T03:14:34.544 DEBUG:teuthology.orchestra.run.smithi037:> do 2022-03-10T03:14:34.545 DEBUG:teuthology.orchestra.run.smithi037:> zap_completion=`kubectl get job -n rook-ceph rook-ceph-device-zap -o jsonpath='"'"'{.status.succeeded.path}'"'"'` 2022-03-10T03:14:34.545 DEBUG:teuthology.orchestra.run.smithi037:> sleep 3s 2022-03-10T03:14:34.545 DEBUG:teuthology.orchestra.run.smithi037:> done 2022-03-10T03:14:34.545 DEBUG:teuthology.orchestra.run.smithi037:> kubectl patch pv $removed_pv -p '"'"'{"spec":{"claimRef": null}}'"'"' 2022-03-10T03:14:34.546 DEBUG:teuthology.orchestra.run.smithi037:> toolbox ceph orch apply osd --all-available-devices 2022-03-10T03:14:34.546 DEBUG:teuthology.orchestra.run.smithi037:> kubectl delete job rook-ceph-device-zap -n rook-ceph 2022-03-10T03:14:34.546 DEBUG:teuthology.orchestra.run.smithi037:> num_osd="0" 2022-03-10T03:14:34.547 DEBUG:teuthology.orchestra.run.smithi037:> while [ "$num_osd" != "$orig_num_osd" ] 2022-03-10T03:14:34.547 DEBUG:teuthology.orchestra.run.smithi037:> do 2022-03-10T03:14:34.547 DEBUG:teuthology.orchestra.run.smithi037:> echo "waiting for osd to come back up" 2022-03-10T03:14:34.548 DEBUG:teuthology.orchestra.run.smithi037:> num_osd=`toolbox ceph osd stat | cut -f3 -d " "` 2022-03-10T03:14:34.548 DEBUG:teuthology.orchestra.run.smithi037:> sleep 30s 2022-03-10T03:14:34.548 DEBUG:teuthology.orchestra.run.smithi037:> done 2022-03-10T03:14:34.548 DEBUG:teuthology.orchestra.run.smithi037:> ' 2022-03-10T03:14:34.566 INFO:teuthology.orchestra.run.smithi037.stderr:++ toolbox ceph osd stat 2022-03-10T03:14:34.567 INFO:teuthology.orchestra.run.smithi037.stderr:++ cut -f3 -d ' ' 2022-03-10T03:14:34.567 INFO:teuthology.orchestra.run.smithi037.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-03-10T03:14:36.935 INFO:teuthology.orchestra.run.smithi037.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-10T03:14:37.466 INFO:teuthology.orchestra.run.smithi037.stderr:+ orig_num_osd=12 2022-03-10T03:14:37.466 INFO:teuthology.orchestra.run.smithi037.stderr:+ toolbox ceph orch osd rm 0 --force 2022-03-10T03:14:37.467 INFO:teuthology.orchestra.run.smithi037.stderr:+ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph orch osd rm 0 --force 2022-03-10T03:14:37.539 INFO:teuthology.orchestra.run.smithi037.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-10T03:14:38.180 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:38.180457 I | ceph-cluster-controller: CR has changed for "rook-ceph". diff= v1.ClusterSpec{ 2022-03-10T03:14:38.180 INFO:tasks.rook.operator.smithi037.stdout: CephVersion: {Image: "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", AllowUnsupported: true}, 2022-03-10T03:14:38.181 INFO:tasks.rook.operator.smithi037.stdout: Storage: v1.StorageScopeSpec{ 2022-03-10T03:14:38.181 INFO:tasks.rook.operator.smithi037.stdout: ... // 3 identical fields 2022-03-10T03:14:38.181 INFO:tasks.rook.operator.smithi037.stdout: Config: nil, 2022-03-10T03:14:38.181 INFO:tasks.rook.operator.smithi037.stdout: Selection: {}, 2022-03-10T03:14:38.182 INFO:tasks.rook.operator.smithi037.stdout: StorageClassDeviceSets: []v1.StorageClassDeviceSet{ 2022-03-10T03:14:38.182 INFO:tasks.rook.operator.smithi037.stdout: ... // 2 identical elements 2022-03-10T03:14:38.182 INFO:tasks.rook.operator.smithi037.stdout: {Name: "smithi037-lv-3", Count: 1, VolumeClaimTemplates: {{ObjectMeta: {Name: "data"}, Spec: {AccessModes: {"ReadWriteOnce"}, Resources: {Requests: {s"storage": {i: {...}, s: "1", Format: "DecimalSI"}}}, VolumeName: "smithi037-lv-3", StorageClassName: &"scratch", ...}}}}, 2022-03-10T03:14:38.182 INFO:tasks.rook.operator.smithi037.stdout: {Name: "smithi037-lv-4", Count: 1, VolumeClaimTemplates: {{ObjectMeta: {Name: "data"}, Spec: {AccessModes: {"ReadWriteOnce"}, Resources: {Requests: {s"storage": {i: {...}, s: "1", Format: "DecimalSI"}}}, VolumeName: "smithi037-lv-4", StorageClassName: &"scratch", ...}}}}, 2022-03-10T03:14:38.183 INFO:tasks.rook.operator.smithi037.stdout:- { 2022-03-10T03:14:38.183 INFO:tasks.rook.operator.smithi037.stdout:- Name: "smithi038-lv-1", 2022-03-10T03:14:38.183 INFO:tasks.rook.operator.smithi037.stdout:- Count: 1, 2022-03-10T03:14:38.183 INFO:tasks.rook.operator.smithi037.stdout:- VolumeClaimTemplates: []v1.PersistentVolumeClaim{{ObjectMeta: v1.ObjectMeta{...}, Spec: v1.PersistentVolumeClaimSpec{...}}}, 2022-03-10T03:14:38.184 INFO:tasks.rook.operator.smithi037.stdout:- }, 2022-03-10T03:14:38.184 INFO:tasks.rook.operator.smithi037.stdout: {Name: "smithi038-lv-2", Count: 1, VolumeClaimTemplates: {{ObjectMeta: {Name: "data"}, Spec: {AccessModes: {"ReadWriteOnce"}, Resources: {Requests: {s"storage": {i: {...}, s: "1", Format: "DecimalSI"}}}, VolumeName: "smithi038-lv-2", StorageClassName: &"scratch", ...}}}}, 2022-03-10T03:14:38.184 INFO:tasks.rook.operator.smithi037.stdout: {Name: "smithi038-lv-3", Count: 1, VolumeClaimTemplates: {{ObjectMeta: {Name: "data"}, Spec: {AccessModes: {"ReadWriteOnce"}, Resources: {Requests: {s"storage": {i: {...}, s: "1", Format: "DecimalSI"}}}, VolumeName: "smithi038-lv-3", StorageClassName: &"scratch", ...}}}}, 2022-03-10T03:14:38.184 INFO:tasks.rook.operator.smithi037.stdout: ... // 5 identical elements 2022-03-10T03:14:38.185 INFO:tasks.rook.operator.smithi037.stdout: }, 2022-03-10T03:14:38.185 INFO:tasks.rook.operator.smithi037.stdout: }, 2022-03-10T03:14:38.185 INFO:tasks.rook.operator.smithi037.stdout: Annotations: nil, 2022-03-10T03:14:38.185 INFO:tasks.rook.operator.smithi037.stdout: Labels: nil, 2022-03-10T03:14:38.185 INFO:tasks.rook.operator.smithi037.stdout: ... // 20 identical fields 2022-03-10T03:14:38.186 INFO:tasks.rook.operator.smithi037.stdout: } 2022-03-10T03:14:38.186 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:38.180612 I | operator: reloading operator's CRDs manager, cancelling all orchestrations! 2022-03-10T03:14:38.186 INFO:tasks.rook.operator.smithi037.stdout:I0310 03:14:38.180909 1 manager.go:148] objectbucket.io/provisioner-manager "msg"="stopping provisioner" "name"="rook-ceph.ceph.rook.io/bucket" "reason"={} 2022-03-10T03:14:38.186 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:38.180926 I | operator: successfully started the controller-runtime manager 2022-03-10T03:14:38.190 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:38.195356 I | op-k8sutil: ROOK_CURRENT_NAMESPACE_ONLY="false" (env var) 2022-03-10T03:14:38.191 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:38.195386 I | operator: watching all namespaces for Ceph CRs 2022-03-10T03:14:38.191 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:38.195426 I | operator: setting up schemes 2022-03-10T03:14:38.194 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:38.198214 I | operator: setting up the controller-runtime manager 2022-03-10T03:14:38.497 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:38.502076 I | operator: looking for admission webhook secret "rook-ceph-admission-controller" 2022-03-10T03:14:38.541 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:38.544905 I | operator: admission webhook secret "rook-ceph-admission-controller" not found. proceeding without the admission controller 2022-03-10T03:14:38.542 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:38.544959 I | ceph-cluster-controller: successfully started 2022-03-10T03:14:38.542 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:38.545052 I | ceph-cluster-controller: enabling hotplug orchestration 2022-03-10T03:14:38.542 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:38.545091 I | ceph-crashcollector-controller: successfully started 2022-03-10T03:14:38.542 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:38.545133 I | ceph-block-pool-controller: successfully started 2022-03-10T03:14:38.543 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:38.545179 I | ceph-object-store-user-controller: successfully started 2022-03-10T03:14:38.543 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:38.545225 I | ceph-object-realm-controller: successfully started 2022-03-10T03:14:38.543 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:38.545256 I | ceph-object-zonegroup-controller: successfully started 2022-03-10T03:14:38.543 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:38.545285 I | ceph-object-zone-controller: successfully started 2022-03-10T03:14:38.544 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:38.545476 I | ceph-object-controller: successfully started 2022-03-10T03:14:38.544 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:38.545551 I | ceph-file-controller: successfully started 2022-03-10T03:14:38.544 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:38.545606 I | ceph-nfs-controller: successfully started 2022-03-10T03:14:38.544 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:38.545659 I | ceph-rbd-mirror-controller: successfully started 2022-03-10T03:14:38.545 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:38.545710 I | ceph-client-controller: successfully started 2022-03-10T03:14:38.545 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:38.545748 I | ceph-filesystem-mirror-controller: successfully started 2022-03-10T03:14:38.546 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:38.545803 I | operator: rook-ceph-operator-config-controller successfully started 2022-03-10T03:14:38.546 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:38.545841 I | ceph-csi: rook-ceph-operator-csi-controller successfully started 2022-03-10T03:14:38.546 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:38.545877 I | op-bucket-prov: rook-ceph-operator-bucket-controller successfully started 2022-03-10T03:14:38.546 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:38.545913 I | ceph-bucket-topic: successfully started 2022-03-10T03:14:38.547 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:38.545943 I | ceph-bucket-notification: successfully started 2022-03-10T03:14:38.547 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:38.545970 I | ceph-bucket-notification: successfully started 2022-03-10T03:14:38.547 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:38.545995 I | ceph-fs-subvolumegroup-controller: successfully started 2022-03-10T03:14:38.549 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:38.547266 I | operator: starting the controller-runtime manager 2022-03-10T03:14:38.654 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:38.658735 I | op-k8sutil: ROOK_CEPH_COMMANDS_TIMEOUT_SECONDS="15" (configmap) 2022-03-10T03:14:38.655 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:38.658774 I | op-k8sutil: ROOK_LOG_LEVEL="INFO" (configmap) 2022-03-10T03:14:38.657 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:38.658785 I | op-k8sutil: ROOK_ENABLE_DISCOVERY_DAEMON="false" (configmap) 2022-03-10T03:14:38.658 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:38.659054 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-03-10T03:14:38.710 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:38.714597 I | operator: rook-ceph-operator-config-controller done reconciling 2022-03-10T03:14:38.743 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:38.747588 I | op-mon: parsing mon endpoints: a=10.100.186.100:6789,b=10.111.132.151:6789,c=10.101.46.100:6789 2022-03-10T03:14:38.743 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:38.747671 I | op-k8sutil: ROOK_OBC_WATCH_OPERATOR_NAMESPACE="true" (configmap) 2022-03-10T03:14:38.744 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:38.747683 I | op-bucket-prov: ceph bucket provisioner launched watching for provisioner "rook-ceph.ceph.rook.io/bucket" 2022-03-10T03:14:38.744 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:38.747734 I | op-mon: parsing mon endpoints: a=10.100.186.100:6789,b=10.111.132.151:6789,c=10.101.46.100:6789 2022-03-10T03:14:38.748 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:38.749065 I | op-bucket-prov: successfully reconciled bucket provisioner 2022-03-10T03:14:38.748 INFO:tasks.rook.operator.smithi037.stdout:I0310 03:14:38.749130 1 manager.go:135] objectbucket.io/provisioner-manager "msg"="starting provisioner" "name"="rook-ceph.ceph.rook.io/bucket" 2022-03-10T03:14:38.770 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:38.775070 I | ceph-spec: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3... 2022-03-10T03:14:38.792 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:38.797085 I | ceph-csi: successfully created csi config map "rook-ceph-csi-config" 2022-03-10T03:14:38.804 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:38.807018 I | op-k8sutil: ROOK_CSI_ENABLE_RBD="true" (configmap) 2022-03-10T03:14:38.804 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:38.807037 I | op-k8sutil: ROOK_CSI_ENABLE_CEPHFS="true" (configmap) 2022-03-10T03:14:38.804 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:38.807042 I | op-k8sutil: ROOK_CSI_ALLOW_UNSUPPORTED_VERSION="false" (configmap) 2022-03-10T03:14:38.805 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:38.807048 I | op-k8sutil: ROOK_CSI_ENABLE_GRPC_METRICS="false" (configmap) 2022-03-10T03:14:38.805 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:38.807053 I | op-k8sutil: CSI_ENABLE_HOST_NETWORK="true" (default) 2022-03-10T03:14:38.805 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:38.807057 I | op-k8sutil: ROOK_CSI_CEPH_IMAGE="quay.io/cephcsi/cephcsi:v3.5.1" (default) 2022-03-10T03:14:38.805 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:38.807066 I | op-k8sutil: ROOK_CSI_REGISTRAR_IMAGE="k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.5.0" (default) 2022-03-10T03:14:38.806 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:38.807074 I | op-k8sutil: ROOK_CSI_PROVISIONER_IMAGE="k8s.gcr.io/sig-storage/csi-provisioner:v3.1.0" (default) 2022-03-10T03:14:38.806 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:38.807079 I | op-k8sutil: ROOK_CSI_ATTACHER_IMAGE="k8s.gcr.io/sig-storage/csi-attacher:v3.4.0" (default) 2022-03-10T03:14:38.806 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:38.807084 I | op-k8sutil: ROOK_CSI_SNAPSHOTTER_IMAGE="k8s.gcr.io/sig-storage/csi-snapshotter:v5.0.1" (default) 2022-03-10T03:14:38.806 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:38.807090 I | op-k8sutil: ROOK_CSI_KUBELET_DIR_PATH="/var/lib/kubelet" (default) 2022-03-10T03:14:38.806 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:38.807100 I | op-k8sutil: CSI_VOLUME_REPLICATION_IMAGE="quay.io/csiaddons/volumereplication-operator:v0.3.0" (default) 2022-03-10T03:14:38.807 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:38.807109 I | op-k8sutil: ROOK_CSIADDONS_IMAGE="quay.io/csiaddons/k8s-sidecar:v0.2.1" (default) 2022-03-10T03:14:38.807 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:38.807113 I | op-k8sutil: ROOK_CSI_CEPHFS_POD_LABELS="" (default) 2022-03-10T03:14:38.807 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:38.807118 I | op-k8sutil: ROOK_CSI_RBD_POD_LABELS="" (default) 2022-03-10T03:14:38.807 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:38.807124 I | ceph-csi: detecting the ceph csi image version for image "quay.io/cephcsi/cephcsi:v3.5.1" 2022-03-10T03:14:38.808 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:38.807164 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-03-10T03:14:38.808 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:38.807176 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-10T03:14:43.245 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:43.249395 I | ceph-spec: object "rook-ceph-osd-0" matched on delete, reconciling 2022-03-10T03:14:44.605 INFO:teuthology.orchestra.run.smithi037.stdout:Success 2022-03-10T03:14:44.628 INFO:teuthology.orchestra.run.smithi037.stderr:+ removed_pv= 2022-03-10T03:14:44.629 INFO:teuthology.orchestra.run.smithi037.stderr:+ '[' '' = '' ']' 2022-03-10T03:14:44.629 INFO:teuthology.orchestra.run.smithi037.stderr:++ kubectl get pv 2022-03-10T03:14:44.630 INFO:teuthology.orchestra.run.smithi037.stderr:++ grep Released 2022-03-10T03:14:44.630 INFO:teuthology.orchestra.run.smithi037.stderr:++ cut -f1 -d ' ' 2022-03-10T03:14:44.696 INFO:teuthology.orchestra.run.smithi037.stderr:+ removed_pv=smithi038-lv-1 2022-03-10T03:14:44.697 INFO:teuthology.orchestra.run.smithi037.stderr:+ sleep 3s 2022-03-10T03:14:45.424 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:45.427913 I | ceph-csi: Detected ceph CSI image version: "v3.5.1" 2022-03-10T03:14:45.427 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:45.427956 I | op-k8sutil: CSI_FORCE_CEPHFS_KERNEL_CLIENT="true" (configmap) 2022-03-10T03:14:45.427 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:45.427967 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default) 2022-03-10T03:14:45.428 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:45.427975 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default) 2022-03-10T03:14:45.428 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:45.427983 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default) 2022-03-10T03:14:45.429 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:45.427993 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default) 2022-03-10T03:14:45.429 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:45.428001 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default) 2022-03-10T03:14:45.429 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:45.428008 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default) 2022-03-10T03:14:45.429 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:45.428016 I | op-k8sutil: CSIADDONS_PORT="9070" (default) 2022-03-10T03:14:45.430 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:45.428023 I | op-k8sutil: CSIADDONS_PORT="9070" (default) 2022-03-10T03:14:45.430 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:45.428031 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default) 2022-03-10T03:14:45.430 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:45.428038 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default) 2022-03-10T03:14:45.431 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:45.428045 I | op-k8sutil: CSI_PLUGIN_PRIORITY_CLASSNAME="" (default) 2022-03-10T03:14:45.431 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:45.428052 I | op-k8sutil: CSI_PROVISIONER_PRIORITY_CLASSNAME="" (default) 2022-03-10T03:14:45.431 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:45.428060 I | op-k8sutil: CSI_ENABLE_OMAP_GENERATOR="false" (default) 2022-03-10T03:14:45.431 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:45.428067 I | op-k8sutil: CSI_ENABLE_RBD_SNAPSHOTTER="true" (configmap) 2022-03-10T03:14:45.432 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:45.428074 I | op-k8sutil: CSI_ENABLE_CEPHFS_SNAPSHOTTER="true" (configmap) 2022-03-10T03:14:45.432 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:45.428081 I | op-k8sutil: CSI_ENABLE_VOLUME_REPLICATION="false" (configmap) 2022-03-10T03:14:45.432 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:45.428089 I | op-k8sutil: CSI_ENABLE_CSIADDONS="false" (configmap) 2022-03-10T03:14:45.432 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:45.428097 I | op-k8sutil: CSI_CEPHFS_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-03-10T03:14:45.432 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:45.428105 I | op-k8sutil: CSI_RBD_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-03-10T03:14:45.433 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:45.428114 I | op-k8sutil: CSI_PLUGIN_ENABLE_SELINUX_HOST_MOUNT="false" (configmap) 2022-03-10T03:14:45.433 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:45.428123 I | ceph-csi: Kubernetes version is 1.23 2022-03-10T03:14:45.433 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:45.428137 I | op-k8sutil: ROOK_CSI_RESIZER_IMAGE="k8s.gcr.io/sig-storage/csi-resizer:v1.4.0" (default) 2022-03-10T03:14:45.433 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:45.428144 I | op-k8sutil: CSI_LOG_LEVEL="" (default) 2022-03-10T03:14:45.440 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:45.445457 I | op-k8sutil: CSI_PROVISIONER_REPLICAS="2" (configmap) 2022-03-10T03:14:45.446 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:45.450758 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-03-10T03:14:45.446 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:45.450777 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-10T03:14:45.447 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:45.450783 I | op-k8sutil: CSI_PLUGIN_TOLERATIONS="" (default) 2022-03-10T03:14:45.447 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:45.450787 I | op-k8sutil: CSI_PLUGIN_NODE_AFFINITY="" (default) 2022-03-10T03:14:45.447 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:45.450791 I | op-k8sutil: CSI_RBD_PLUGIN_TOLERATIONS="" (default) 2022-03-10T03:14:45.447 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:45.450795 I | op-k8sutil: CSI_RBD_PLUGIN_NODE_AFFINITY="" (default) 2022-03-10T03:14:45.448 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:45.450799 I | op-k8sutil: CSI_RBD_PLUGIN_RESOURCE="" (default) 2022-03-10T03:14:46.006 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:46.004464 I | op-k8sutil: CSI_RBD_PROVISIONER_TOLERATIONS="" (default) 2022-03-10T03:14:46.006 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:46.004500 I | op-k8sutil: CSI_RBD_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-10T03:14:46.007 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:46.004510 I | op-k8sutil: CSI_RBD_PROVISIONER_RESOURCE="" (default) 2022-03-10T03:14:46.411 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:46.415273 I | ceph-csi: successfully started CSI Ceph RBD driver 2022-03-10T03:14:46.829 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:46.833514 I | op-k8sutil: CSI_CEPHFS_PLUGIN_TOLERATIONS="" (default) 2022-03-10T03:14:46.830 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:46.833552 I | op-k8sutil: CSI_CEPHFS_PLUGIN_NODE_AFFINITY="" (default) 2022-03-10T03:14:46.831 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:46.833563 I | op-k8sutil: CSI_CEPHFS_PLUGIN_RESOURCE="" (default) 2022-03-10T03:14:46.970 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:46.974496 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_TOLERATIONS="" (default) 2022-03-10T03:14:46.971 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:46.974526 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-10T03:14:46.971 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:46.974535 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_RESOURCE="" (default) 2022-03-10T03:14:47.007 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:47.011981 I | ceph-spec: detected ceph image version: "17.0.0-11025 quincy" 2022-03-10T03:14:47.008 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:47.011998 I | ceph-cluster-controller: validating ceph version from provided image 2022-03-10T03:14:47.008 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:47.012006 W | ceph-cluster-controller: unsupported ceph version detected: "17.0.0-11025 quincy", pursuing 2022-03-10T03:14:47.021 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:47.025687 I | op-mon: parsing mon endpoints: a=10.100.186.100:6789,b=10.111.132.151:6789,c=10.101.46.100:6789 2022-03-10T03:14:47.268 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:47.272607 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-10T03:14:47.272 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:47.273610 I | ceph-csi: successfully started CSI CephFS driver 2022-03-10T03:14:47.447 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:47.451349 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-10T03:14:47.699 INFO:teuthology.orchestra.run.smithi037.stderr:+ '[' smithi038-lv-1 = '' ']' 2022-03-10T03:14:47.700 INFO:teuthology.orchestra.run.smithi037.stderr:++ kubectl get pv smithi038-lv-1 -o 'jsonpath={.spec.local.path}' 2022-03-10T03:14:47.753 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:47.757336 I | op-k8sutil: CSI_RBD_FSGROUPPOLICY="ReadWriteOnceWithFSType" (configmap) 2022-03-10T03:14:47.755 INFO:teuthology.orchestra.run.smithi037.stderr:+ target_path=/dev/vg_nvme/lv_1 2022-03-10T03:14:47.756 INFO:teuthology.orchestra.run.smithi037.stderr:++ echo smithi038-lv-1 2022-03-10T03:14:47.756 INFO:teuthology.orchestra.run.smithi037.stderr:++ cut -f1 -d - 2022-03-10T03:14:47.757 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:47.761349 I | ceph-csi: CSIDriver object updated for driver "rook-ceph.rbd.csi.ceph.com" 2022-03-10T03:14:47.757 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:47.761378 I | op-k8sutil: CSI_CEPHFS_FSGROUPPOLICY="ReadWriteOnceWithFSType" (configmap) 2022-03-10T03:14:47.758 INFO:teuthology.orchestra.run.smithi037.stderr:+ host=smithi038 2022-03-10T03:14:47.758 INFO:teuthology.orchestra.run.smithi037.stderr:+ toolbox ceph orch device zap smithi038 /dev/vg_nvme/lv_1 --force 2022-03-10T03:14:47.759 INFO:teuthology.orchestra.run.smithi037.stderr:+ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph orch device zap smithi038 /dev/vg_nvme/lv_1 --force 2022-03-10T03:14:47.762 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:47.765801 I | ceph-csi: CSIDriver object updated for driver "rook-ceph.cephfs.csi.ceph.com" 2022-03-10T03:14:47.832 INFO:teuthology.orchestra.run.smithi037.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-10T03:14:47.884 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:47.888738 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-11025 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3" 2022-03-10T03:14:48.140 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:48.144508 I | op-mon: start running mons 2022-03-10T03:14:48.164 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:48.168785 I | op-mon: parsing mon endpoints: a=10.100.186.100:6789,b=10.111.132.151:6789,c=10.101.46.100:6789 2022-03-10T03:14:48.374 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:48.379206 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.100.186.100:6789","10.111.132.151:6789","10.101.46.100:6789"]}] data:a=10.100.186.100:6789,b=10.111.132.151:6789,c=10.101.46.100:6789 mapping:{"node":{"a":{"Name":"smithi038","Hostname":"smithi038","Address":"172.21.15.38"},"b":{"Name":"smithi195","Hostname":"smithi195","Address":"172.21.15.195"},"c":{"Name":"smithi037","Hostname":"smithi037","Address":"172.21.15.37"}}} maxMonId:2] 2022-03-10T03:14:48.377 INFO:teuthology.orchestra.run.smithi037.stdout:/dev/vg_nvme/lv_1 on smithi038 zapped 2022-03-10T03:14:48.402 INFO:teuthology.orchestra.run.smithi037.stderr:+ zap_completion=0 2022-03-10T03:14:48.402 INFO:teuthology.orchestra.run.smithi037.stderr:+ '[' 0 = 0 ']' 2022-03-10T03:14:48.402 INFO:teuthology.orchestra.run.smithi037.stderr:++ kubectl get job -n rook-ceph rook-ceph-device-zap -o 'jsonpath={.status.succeeded.path}' 2022-03-10T03:14:48.557 INFO:teuthology.orchestra.run.smithi037.stderr:+ zap_completion= 2022-03-10T03:14:48.558 INFO:teuthology.orchestra.run.smithi037.stderr:+ sleep 3s 2022-03-10T03:14:49.099 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:49.103322 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-10T03:14:49.144 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:49.148496 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-10T03:14:50.581 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:50.585806 I | op-mon: targeting the mon count 3 2022-03-10T03:14:50.598 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:50.594184 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-10T03:14:50.982 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:50.986430 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-10T03:14:50.982 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:50.986456 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-03-10T03:14:51.364 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:51.368703 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-03-10T03:14:51.365 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:51.368744 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-10T03:14:51.558 INFO:teuthology.orchestra.run.smithi037.stderr:+ '[' '' = 0 ']' 2022-03-10T03:14:51.559 INFO:teuthology.orchestra.run.smithi037.stderr:+ kubectl patch pv smithi038-lv-1 -p '{"spec":{"claimRef": null}}' 2022-03-10T03:14:51.636 INFO:teuthology.orchestra.run.smithi037.stdout:persistentvolume/smithi038-lv-1 patched 2022-03-10T03:14:51.639 INFO:teuthology.orchestra.run.smithi037.stderr:+ toolbox ceph orch apply osd --all-available-devices 2022-03-10T03:14:51.639 INFO:teuthology.orchestra.run.smithi037.stderr:+ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph orch apply osd --all-available-devices 2022-03-10T03:14:51.709 INFO:teuthology.orchestra.run.smithi037.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-10T03:14:51.788 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:51.792531 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-10T03:14:51.788 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:51.792564 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-03-10T03:14:52.393 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:52.397170 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-03-10T03:14:52.394 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:52.397202 I | op-config: deleting "log file" option from the mon configuration database 2022-03-10T03:14:52.430 INFO:teuthology.orchestra.run.smithi037.stdout:['Success'] 2022-03-10T03:14:52.436 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:52.438146 I | ceph-cluster-controller: CR has changed for "rook-ceph". diff= v1.ClusterSpec{ 2022-03-10T03:14:52.436 INFO:tasks.rook.operator.smithi037.stdout: CephVersion: {Image: "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", AllowUnsupported: true}, 2022-03-10T03:14:52.436 INFO:tasks.rook.operator.smithi037.stdout: Storage: v1.StorageScopeSpec{ 2022-03-10T03:14:52.436 INFO:tasks.rook.operator.smithi037.stdout: ... // 3 identical fields 2022-03-10T03:14:52.437 INFO:tasks.rook.operator.smithi037.stdout: Config: nil, 2022-03-10T03:14:52.437 INFO:tasks.rook.operator.smithi037.stdout: Selection: {}, 2022-03-10T03:14:52.437 INFO:tasks.rook.operator.smithi037.stdout: StorageClassDeviceSets: []v1.StorageClassDeviceSet{ 2022-03-10T03:14:52.437 INFO:tasks.rook.operator.smithi037.stdout: ... // 9 identical elements 2022-03-10T03:14:52.438 INFO:tasks.rook.operator.smithi037.stdout: {Name: "smithi195-lv-3", Count: 1, VolumeClaimTemplates: {{ObjectMeta: {Name: "data"}, Spec: {AccessModes: {"ReadWriteOnce"}, Resources: {Requests: {s"storage": {i: {...}, s: "1", Format: "DecimalSI"}}}, VolumeName: "smithi195-lv-3", StorageClassName: &"scratch", ...}}}}, 2022-03-10T03:14:52.438 INFO:tasks.rook.operator.smithi037.stdout: {Name: "smithi195-lv-4", Count: 1, VolumeClaimTemplates: {{ObjectMeta: {Name: "data"}, Spec: {AccessModes: {"ReadWriteOnce"}, Resources: {Requests: {s"storage": {i: {...}, s: "1", Format: "DecimalSI"}}}, VolumeName: "smithi195-lv-4", StorageClassName: &"scratch", ...}}}}, 2022-03-10T03:14:52.438 INFO:tasks.rook.operator.smithi037.stdout:+ { 2022-03-10T03:14:52.438 INFO:tasks.rook.operator.smithi037.stdout:+ Name: "smithi038-lv-1", 2022-03-10T03:14:52.439 INFO:tasks.rook.operator.smithi037.stdout:+ Count: 1, 2022-03-10T03:14:52.439 INFO:tasks.rook.operator.smithi037.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{ObjectMeta: v1.ObjectMeta{...}, Spec: v1.PersistentVolumeClaimSpec{...}}}, 2022-03-10T03:14:52.439 INFO:tasks.rook.operator.smithi037.stdout:+ }, 2022-03-10T03:14:52.439 INFO:tasks.rook.operator.smithi037.stdout: }, 2022-03-10T03:14:52.439 INFO:tasks.rook.operator.smithi037.stdout: }, 2022-03-10T03:14:52.440 INFO:tasks.rook.operator.smithi037.stdout: Annotations: nil, 2022-03-10T03:14:52.440 INFO:tasks.rook.operator.smithi037.stdout: Labels: nil, 2022-03-10T03:14:52.440 INFO:tasks.rook.operator.smithi037.stdout: ... // 20 identical fields 2022-03-10T03:14:52.440 INFO:tasks.rook.operator.smithi037.stdout: } 2022-03-10T03:14:52.441 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:52.438278 I | operator: reloading operator's CRDs manager, cancelling all orchestrations! 2022-03-10T03:14:52.441 INFO:tasks.rook.operator.smithi037.stdout:I0310 03:14:52.438536 1 manager.go:148] objectbucket.io/provisioner-manager "msg"="stopping provisioner" "name"="rook-ceph.ceph.rook.io/bucket" "reason"={} 2022-03-10T03:14:52.455 INFO:teuthology.orchestra.run.smithi037.stderr:+ kubectl delete job rook-ceph-device-zap -n rook-ceph 2022-03-10T03:14:52.492 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:52.496098 I | op-k8sutil: ROOK_CURRENT_NAMESPACE_ONLY="false" (env var) 2022-03-10T03:14:52.492 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:52.496131 I | operator: watching all namespaces for Ceph CRs 2022-03-10T03:14:52.492 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:52.496185 I | operator: setting up schemes 2022-03-10T03:14:52.497 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:52.501695 I | operator: setting up the controller-runtime manager 2022-03-10T03:14:52.519 INFO:teuthology.orchestra.run.smithi037.stdout:job.batch "rook-ceph-device-zap" deleted 2022-03-10T03:14:52.531 INFO:teuthology.orchestra.run.smithi037.stderr:+ num_osd=0 2022-03-10T03:14:52.532 INFO:teuthology.orchestra.run.smithi037.stderr:+ '[' 0 '!=' 12 ']' 2022-03-10T03:14:52.532 INFO:teuthology.orchestra.run.smithi037.stderr:+ echo 'waiting for osd to come back up' 2022-03-10T03:14:52.532 INFO:teuthology.orchestra.run.smithi037.stdout:waiting for osd to come back up 2022-03-10T03:14:52.533 INFO:teuthology.orchestra.run.smithi037.stderr:++ cut -f3 -d ' ' 2022-03-10T03:14:52.533 INFO:teuthology.orchestra.run.smithi037.stderr:++ toolbox ceph osd stat 2022-03-10T03:14:52.534 INFO:teuthology.orchestra.run.smithi037.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-03-10T03:14:52.624 INFO:teuthology.orchestra.run.smithi037.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-10T03:14:52.798 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:52.801578 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-03-10T03:14:52.798 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:52.801610 I | op-mon: checking for basic quorum with existing mons 2022-03-10T03:14:52.802 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:52.806216 I | operator: looking for admission webhook secret "rook-ceph-admission-controller" 2022-03-10T03:14:52.806 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:52.809434 I | operator: admission webhook secret "rook-ceph-admission-controller" not found. proceeding without the admission controller 2022-03-10T03:14:52.807 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:52.809476 I | ceph-cluster-controller: successfully started 2022-03-10T03:14:52.807 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:52.809543 I | ceph-cluster-controller: enabling hotplug orchestration 2022-03-10T03:14:52.808 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:52.809569 I | ceph-crashcollector-controller: successfully started 2022-03-10T03:14:52.809 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:52.809602 I | ceph-block-pool-controller: successfully started 2022-03-10T03:14:52.810 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:52.809631 I | ceph-object-store-user-controller: successfully started 2022-03-10T03:14:52.810 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:52.809660 I | ceph-object-realm-controller: successfully started 2022-03-10T03:14:52.810 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:52.809679 I | ceph-object-zonegroup-controller: successfully started 2022-03-10T03:14:52.810 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:52.809699 I | ceph-object-zone-controller: successfully started 2022-03-10T03:14:52.811 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:52.809832 I | ceph-object-controller: successfully started 2022-03-10T03:14:52.811 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:52.809877 I | ceph-file-controller: successfully started 2022-03-10T03:14:52.811 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:52.809920 I | ceph-nfs-controller: successfully started 2022-03-10T03:14:52.811 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:52.809955 I | ceph-rbd-mirror-controller: successfully started 2022-03-10T03:14:52.812 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:52.809990 I | ceph-client-controller: successfully started 2022-03-10T03:14:52.812 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:52.810020 I | ceph-filesystem-mirror-controller: successfully started 2022-03-10T03:14:52.812 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:52.810056 I | operator: rook-ceph-operator-config-controller successfully started 2022-03-10T03:14:52.812 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:52.810080 I | ceph-csi: rook-ceph-operator-csi-controller successfully started 2022-03-10T03:14:52.812 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:52.810105 I | op-bucket-prov: rook-ceph-operator-bucket-controller successfully started 2022-03-10T03:14:52.813 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:52.810130 I | ceph-bucket-topic: successfully started 2022-03-10T03:14:52.813 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:52.810152 I | ceph-bucket-notification: successfully started 2022-03-10T03:14:52.813 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:52.810171 I | ceph-bucket-notification: successfully started 2022-03-10T03:14:52.813 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:52.810193 I | ceph-fs-subvolumegroup-controller: successfully started 2022-03-10T03:14:52.814 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:52.811332 I | operator: starting the controller-runtime manager 2022-03-10T03:14:52.818 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:52.823169 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". failed to reconcile cluster "rook-ceph": failed to configure local ceph cluster: failed to create cluster: failed to start ceph monitors: failed to init mon services: context canceled 2022-03-10T03:14:52.823 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:52.826184 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-03-10T03:14:52.845 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:52.850236 I | op-mon: parsing mon endpoints: a=10.100.186.100:6789,b=10.111.132.151:6789,c=10.101.46.100:6789 2022-03-10T03:14:52.862 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:52.866709 E | ceph-spec: failed to update cluster condition to {Type:Progressing Status:True Reason:ClusterProgressing Message:Detecting Ceph version LastHeartbeatTime:2022-03-10 03:14:52.858790094 +0000 UTC m=+468.552624191 LastTransitionTime:2022-03-10 03:14:52.858789842 +0000 UTC m=+468.552623964}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-10T03:14:52.862 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:52.866735 I | ceph-spec: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3... 2022-03-10T03:14:52.909 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:52.912859 I | op-k8sutil: ROOK_CEPH_COMMANDS_TIMEOUT_SECONDS="15" (configmap) 2022-03-10T03:14:52.909 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:52.912924 I | op-k8sutil: ROOK_LOG_LEVEL="INFO" (configmap) 2022-03-10T03:14:52.909 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:52.912936 I | op-k8sutil: ROOK_ENABLE_DISCOVERY_DAEMON="false" (configmap) 2022-03-10T03:14:52.912 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:52.917115 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-03-10T03:14:52.972 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:52.976739 I | operator: rook-ceph-operator-config-controller done reconciling 2022-03-10T03:14:52.998 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:53.001918 I | op-mon: parsing mon endpoints: a=10.100.186.100:6789,b=10.111.132.151:6789,c=10.101.46.100:6789 2022-03-10T03:14:52.998 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:53.001969 I | op-k8sutil: ROOK_OBC_WATCH_OPERATOR_NAMESPACE="true" (configmap) 2022-03-10T03:14:52.998 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:53.001978 I | op-bucket-prov: ceph bucket provisioner launched watching for provisioner "rook-ceph.ceph.rook.io/bucket" 2022-03-10T03:14:52.998 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:53.002002 I | op-mon: parsing mon endpoints: a=10.100.186.100:6789,b=10.111.132.151:6789,c=10.101.46.100:6789 2022-03-10T03:14:53.000 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:53.002997 I | op-bucket-prov: successfully reconciled bucket provisioner 2022-03-10T03:14:53.000 INFO:tasks.rook.operator.smithi037.stdout:I0310 03:14:53.003051 1 manager.go:135] objectbucket.io/provisioner-manager "msg"="starting provisioner" "name"="rook-ceph.ceph.rook.io/bucket" 2022-03-10T03:14:53.029 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:53.033837 I | ceph-csi: successfully created csi config map "rook-ceph-csi-config" 2022-03-10T03:14:53.044 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:53.048690 I | ceph-spec: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3... 2022-03-10T03:14:53.117 INFO:teuthology.orchestra.run.smithi037.stderr:+ num_osd=11 2022-03-10T03:14:53.117 INFO:teuthology.orchestra.run.smithi037.stderr:+ sleep 30s 2022-03-10T03:14:53.214 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:53.217651 I | op-k8sutil: ROOK_CSI_ENABLE_RBD="true" (configmap) 2022-03-10T03:14:53.215 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:53.217674 I | op-k8sutil: ROOK_CSI_ENABLE_CEPHFS="true" (configmap) 2022-03-10T03:14:53.215 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:53.217680 I | op-k8sutil: ROOK_CSI_ALLOW_UNSUPPORTED_VERSION="false" (configmap) 2022-03-10T03:14:53.215 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:53.217687 I | op-k8sutil: ROOK_CSI_ENABLE_GRPC_METRICS="false" (configmap) 2022-03-10T03:14:53.215 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:53.217693 I | op-k8sutil: CSI_ENABLE_HOST_NETWORK="true" (default) 2022-03-10T03:14:53.216 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:53.217698 I | op-k8sutil: ROOK_CSI_CEPH_IMAGE="quay.io/cephcsi/cephcsi:v3.5.1" (default) 2022-03-10T03:14:53.216 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:53.217705 I | op-k8sutil: ROOK_CSI_REGISTRAR_IMAGE="k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.5.0" (default) 2022-03-10T03:14:53.217 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:53.217710 I | op-k8sutil: ROOK_CSI_PROVISIONER_IMAGE="k8s.gcr.io/sig-storage/csi-provisioner:v3.1.0" (default) 2022-03-10T03:14:53.217 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:53.217716 I | op-k8sutil: ROOK_CSI_ATTACHER_IMAGE="k8s.gcr.io/sig-storage/csi-attacher:v3.4.0" (default) 2022-03-10T03:14:53.218 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:53.217721 I | op-k8sutil: ROOK_CSI_SNAPSHOTTER_IMAGE="k8s.gcr.io/sig-storage/csi-snapshotter:v5.0.1" (default) 2022-03-10T03:14:53.218 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:53.217726 I | op-k8sutil: ROOK_CSI_KUBELET_DIR_PATH="/var/lib/kubelet" (default) 2022-03-10T03:14:53.218 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:53.217732 I | op-k8sutil: CSI_VOLUME_REPLICATION_IMAGE="quay.io/csiaddons/volumereplication-operator:v0.3.0" (default) 2022-03-10T03:14:53.218 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:53.217739 I | op-k8sutil: ROOK_CSIADDONS_IMAGE="quay.io/csiaddons/k8s-sidecar:v0.2.1" (default) 2022-03-10T03:14:53.218 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:53.217743 I | op-k8sutil: ROOK_CSI_CEPHFS_POD_LABELS="" (default) 2022-03-10T03:14:53.219 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:53.217748 I | op-k8sutil: ROOK_CSI_RBD_POD_LABELS="" (default) 2022-03-10T03:14:53.219 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:53.217754 I | ceph-csi: detecting the ceph csi image version for image "quay.io/cephcsi/cephcsi:v3.5.1" 2022-03-10T03:14:53.220 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:53.217808 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-03-10T03:14:53.220 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:53.217818 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-10T03:14:53.410 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:53.414771 I | op-k8sutil: Removing previous job rook-ceph-detect-version to start a new one 2022-03-10T03:14:53.472 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:53.476923 I | op-k8sutil: batch job rook-ceph-detect-version still exists 2022-03-10T03:14:56.479 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:56.483716 I | op-k8sutil: batch job rook-ceph-detect-version still exists 2022-03-10T03:14:59.486 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:14:59.490485 I | op-k8sutil: batch job rook-ceph-detect-version still exists 2022-03-10T03:15:01.063 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:01.064065 I | ceph-csi: Detected ceph CSI image version: "v3.5.1" 2022-03-10T03:15:01.063 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:01.064109 I | op-k8sutil: CSI_FORCE_CEPHFS_KERNEL_CLIENT="true" (configmap) 2022-03-10T03:15:01.063 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:01.064124 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default) 2022-03-10T03:15:01.064 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:01.064131 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default) 2022-03-10T03:15:01.064 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:01.064141 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default) 2022-03-10T03:15:01.064 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:01.064148 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default) 2022-03-10T03:15:01.064 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:01.064160 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default) 2022-03-10T03:15:01.065 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:01.064172 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default) 2022-03-10T03:15:01.065 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:01.064186 I | op-k8sutil: CSIADDONS_PORT="9070" (default) 2022-03-10T03:15:01.065 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:01.064198 I | op-k8sutil: CSIADDONS_PORT="9070" (default) 2022-03-10T03:15:01.065 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:01.064210 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default) 2022-03-10T03:15:01.065 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:01.064222 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default) 2022-03-10T03:15:01.066 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:01.064234 I | op-k8sutil: CSI_PLUGIN_PRIORITY_CLASSNAME="" (default) 2022-03-10T03:15:01.066 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:01.064245 I | op-k8sutil: CSI_PROVISIONER_PRIORITY_CLASSNAME="" (default) 2022-03-10T03:15:01.066 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:01.064257 I | op-k8sutil: CSI_ENABLE_OMAP_GENERATOR="false" (default) 2022-03-10T03:15:01.066 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:01.064269 I | op-k8sutil: CSI_ENABLE_RBD_SNAPSHOTTER="true" (configmap) 2022-03-10T03:15:01.067 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:01.064296 I | op-k8sutil: CSI_ENABLE_CEPHFS_SNAPSHOTTER="true" (configmap) 2022-03-10T03:15:01.067 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:01.064306 I | op-k8sutil: CSI_ENABLE_VOLUME_REPLICATION="false" (configmap) 2022-03-10T03:15:01.067 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:01.064317 I | op-k8sutil: CSI_ENABLE_CSIADDONS="false" (configmap) 2022-03-10T03:15:01.068 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:01.064329 I | op-k8sutil: CSI_CEPHFS_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-03-10T03:15:01.068 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:01.064343 I | op-k8sutil: CSI_RBD_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-03-10T03:15:01.068 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:01.064355 I | op-k8sutil: CSI_PLUGIN_ENABLE_SELINUX_HOST_MOUNT="false" (configmap) 2022-03-10T03:15:01.069 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:01.064365 I | ceph-csi: Kubernetes version is 1.23 2022-03-10T03:15:01.069 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:01.064379 I | op-k8sutil: ROOK_CSI_RESIZER_IMAGE="k8s.gcr.io/sig-storage/csi-resizer:v1.4.0" (default) 2022-03-10T03:15:01.069 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:01.064389 I | op-k8sutil: CSI_LOG_LEVEL="" (default) 2022-03-10T03:15:01.105 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:01.108919 I | op-k8sutil: CSI_PROVISIONER_REPLICAS="2" (configmap) 2022-03-10T03:15:01.113 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:01.117842 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-03-10T03:15:01.114 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:01.117867 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-10T03:15:01.114 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:01.117873 I | op-k8sutil: CSI_PLUGIN_TOLERATIONS="" (default) 2022-03-10T03:15:01.114 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:01.117878 I | op-k8sutil: CSI_PLUGIN_NODE_AFFINITY="" (default) 2022-03-10T03:15:01.115 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:01.117882 I | op-k8sutil: CSI_RBD_PLUGIN_TOLERATIONS="" (default) 2022-03-10T03:15:01.115 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:01.117886 I | op-k8sutil: CSI_RBD_PLUGIN_NODE_AFFINITY="" (default) 2022-03-10T03:15:01.115 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:01.117891 I | op-k8sutil: CSI_RBD_PLUGIN_RESOURCE="" (default) 2022-03-10T03:15:01.235 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:01.239406 I | op-k8sutil: CSI_RBD_PROVISIONER_TOLERATIONS="" (default) 2022-03-10T03:15:01.235 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:01.239438 I | op-k8sutil: CSI_RBD_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-10T03:15:01.235 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:01.239447 I | op-k8sutil: CSI_RBD_PROVISIONER_RESOURCE="" (default) 2022-03-10T03:15:01.278 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:01.283192 I | ceph-csi: successfully started CSI Ceph RBD driver 2022-03-10T03:15:01.439 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:01.443439 I | op-k8sutil: CSI_CEPHFS_PLUGIN_TOLERATIONS="" (default) 2022-03-10T03:15:01.439 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:01.443472 I | op-k8sutil: CSI_CEPHFS_PLUGIN_NODE_AFFINITY="" (default) 2022-03-10T03:15:01.439 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:01.443487 I | op-k8sutil: CSI_CEPHFS_PLUGIN_RESOURCE="" (default) 2022-03-10T03:15:01.556 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:01.560306 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_TOLERATIONS="" (default) 2022-03-10T03:15:01.556 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:01.560344 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-10T03:15:01.557 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:01.560355 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_RESOURCE="" (default) 2022-03-10T03:15:01.609 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:01.613282 I | ceph-csi: successfully started CSI CephFS driver 2022-03-10T03:15:01.816 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:01.820876 I | op-k8sutil: CSI_RBD_FSGROUPPOLICY="ReadWriteOnceWithFSType" (configmap) 2022-03-10T03:15:01.879 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:01.883708 I | ceph-csi: CSIDriver object updated for driver "rook-ceph.rbd.csi.ceph.com" 2022-03-10T03:15:01.880 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:01.883737 I | op-k8sutil: CSI_CEPHFS_FSGROUPPOLICY="ReadWriteOnceWithFSType" (configmap) 2022-03-10T03:15:01.884 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:01.889273 I | ceph-csi: CSIDriver object updated for driver "rook-ceph.cephfs.csi.ceph.com" 2022-03-10T03:15:02.492 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:02.497005 I | op-k8sutil: batch job rook-ceph-detect-version deleted 2022-03-10T03:15:08.379 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:08.382818 I | ceph-spec: detected ceph image version: "17.0.0-11025 quincy" 2022-03-10T03:15:08.379 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:08.382861 I | ceph-cluster-controller: validating ceph version from provided image 2022-03-10T03:15:08.379 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:08.382885 W | ceph-cluster-controller: unsupported ceph version detected: "17.0.0-11025 quincy", pursuing 2022-03-10T03:15:08.395 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:08.398688 I | ceph-spec: detected ceph image version: "17.0.0-11025 quincy" 2022-03-10T03:15:08.395 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:08.398713 I | ceph-cluster-controller: validating ceph version from provided image 2022-03-10T03:15:08.396 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:08.398721 W | ceph-cluster-controller: unsupported ceph version detected: "17.0.0-11025 quincy", pursuing 2022-03-10T03:15:08.424 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:08.428266 I | op-mon: parsing mon endpoints: a=10.100.186.100:6789,b=10.111.132.151:6789,c=10.101.46.100:6789 2022-03-10T03:15:08.470 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:08.474586 I | op-mon: parsing mon endpoints: a=10.100.186.100:6789,b=10.111.132.151:6789,c=10.101.46.100:6789 2022-03-10T03:15:08.471 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:08.474763 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-10T03:15:08.471 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:08.475105 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-10T03:15:08.475 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:08.477738 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-10T03:15:08.476 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:08.477944 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-10T03:15:08.897 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:08.901756 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-11025 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3" 2022-03-10T03:15:09.054 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:08.910251 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-11025 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3" 2022-03-10T03:15:09.055 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:08.960060 E | ceph-spec: failed to update cluster condition to {Type:Progressing Status:True Reason:ClusterProgressing Message:Configuring the Ceph cluster LastHeartbeatTime:2022-03-10 03:15:08.930198264 +0000 UTC m=+484.624032354 LastTransitionTime:2022-03-10 03:15:08.930198044 +0000 UTC m=+484.624032166}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-10T03:15:09.055 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:09.031931 I | op-mon: start running mons 2022-03-10T03:15:09.055 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:09.054143 E | ceph-spec: failed to update cluster condition to {Type:Progressing Status:True Reason:ClusterProgressing Message:Configuring Ceph Mons LastHeartbeatTime:2022-03-10 03:15:09.016282468 +0000 UTC m=+484.710116563 LastTransitionTime:2022-03-10 03:15:09.016282184 +0000 UTC m=+484.710116340}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-10T03:15:09.138 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:09.141662 I | op-mon: parsing mon endpoints: a=10.100.186.100:6789,b=10.111.132.151:6789,c=10.101.46.100:6789 2022-03-10T03:15:10.140 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:10.145004 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.111.132.151:6789","10.101.46.100:6789","10.100.186.100:6789"]}] data:a=10.100.186.100:6789,b=10.111.132.151:6789,c=10.101.46.100:6789 mapping:{"node":{"a":{"Name":"smithi038","Hostname":"smithi038","Address":"172.21.15.38"},"b":{"Name":"smithi195","Hostname":"smithi195","Address":"172.21.15.195"},"c":{"Name":"smithi037","Hostname":"smithi037","Address":"172.21.15.37"}}} maxMonId:2] 2022-03-10T03:15:10.738 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:10.742216 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-10T03:15:10.738 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:10.742453 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-10T03:15:12.367 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:12.371200 I | op-mon: targeting the mon count 3 2022-03-10T03:15:12.451 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:12.455259 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-10T03:15:12.803 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:12.807238 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-10T03:15:12.803 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:12.807268 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-03-10T03:15:13.164 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:13.168192 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-03-10T03:15:13.164 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:13.168228 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-10T03:15:13.507 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:13.511276 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-10T03:15:13.507 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:13.511308 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-03-10T03:15:13.941 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:13.945797 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-03-10T03:15:13.942 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:13.945826 I | op-config: deleting "log file" option from the mon configuration database 2022-03-10T03:15:14.303 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:14.307734 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-03-10T03:15:14.304 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:14.307758 I | op-mon: checking for basic quorum with existing mons 2022-03-10T03:15:14.451 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:14.455281 I | op-mon: mon "a" endpoint is [v2:10.100.186.100:3300,v1:10.100.186.100:6789] 2022-03-10T03:15:14.519 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:14.523104 I | op-mon: mon "b" endpoint is [v2:10.111.132.151:3300,v1:10.111.132.151:6789] 2022-03-10T03:15:14.741 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:14.745766 I | op-mon: mon "c" endpoint is [v2:10.101.46.100:3300,v1:10.101.46.100:6789] 2022-03-10T03:15:15.352 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:15.356826 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.100.186.100:6789","10.111.132.151:6789","10.101.46.100:6789"]}] data:a=10.100.186.100:6789,b=10.111.132.151:6789,c=10.101.46.100:6789 mapping:{"node":{"a":{"Name":"smithi038","Hostname":"smithi038","Address":"172.21.15.38"},"b":{"Name":"smithi195","Hostname":"smithi195","Address":"172.21.15.195"},"c":{"Name":"smithi037","Hostname":"smithi037","Address":"172.21.15.37"}}} maxMonId:2] 2022-03-10T03:15:15.940 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:15.944031 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-10T03:15:15.944 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:15.944360 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-10T03:15:16.364 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:16.368576 I | op-mon: deployment for mon rook-ceph-mon-a already exists. updating if needed 2022-03-10T03:15:16.373 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:16.377748 I | op-k8sutil: deployment "rook-ceph-mon-a" did not change, nothing to update 2022-03-10T03:15:16.373 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:16.377777 I | op-mon: waiting for mon quorum with [a b c] 2022-03-10T03:15:16.967 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:16.971648 I | op-mon: mons running: [a b c] 2022-03-10T03:15:17.408 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:17.412151 I | op-mon: Monitors in quorum: [a b c] 2022-03-10T03:15:17.428 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:17.433326 I | op-mon: deployment for mon rook-ceph-mon-b already exists. updating if needed 2022-03-10T03:15:17.440 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:17.444606 I | op-k8sutil: deployment "rook-ceph-mon-b" did not change, nothing to update 2022-03-10T03:15:17.440 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:17.444635 I | op-mon: waiting for mon quorum with [a b c] 2022-03-10T03:15:17.548 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:17.552826 I | op-mon: mons running: [a b c] 2022-03-10T03:15:17.981 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:17.985976 I | op-mon: Monitors in quorum: [a b c] 2022-03-10T03:15:17.990 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:17.995262 I | op-mon: deployment for mon rook-ceph-mon-c already exists. updating if needed 2022-03-10T03:15:18.002 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:18.006587 I | op-k8sutil: deployment "rook-ceph-mon-c" did not change, nothing to update 2022-03-10T03:15:18.002 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:18.006616 I | op-mon: waiting for mon quorum with [a b c] 2022-03-10T03:15:18.147 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:18.151031 I | op-mon: mons running: [a b c] 2022-03-10T03:15:18.574 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:18.578670 I | op-mon: Monitors in quorum: [a b c] 2022-03-10T03:15:18.575 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:18.578700 I | op-mon: mons created: 3 2022-03-10T03:15:18.980 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:18.984146 I | op-mon: waiting for mon quorum with [a b c] 2022-03-10T03:15:19.019 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:19.023510 I | op-mon: mons running: [a b c] 2022-03-10T03:15:19.440 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:19.443927 I | op-mon: Monitors in quorum: [a b c] 2022-03-10T03:15:19.440 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:19.443988 I | cephclient: getting or creating ceph auth key "client.csi-rbd-provisioner" 2022-03-10T03:15:19.440 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:19.444082 I | op-mon: start running mons 2022-03-10T03:15:19.446 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:19.449483 I | op-mon: parsing mon endpoints: a=10.100.186.100:6789,b=10.111.132.151:6789,c=10.101.46.100:6789 2022-03-10T03:15:19.863 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:19.867324 I | cephclient: getting or creating ceph auth key "client.csi-rbd-node" 2022-03-10T03:15:20.140 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:20.144722 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.100.186.100:6789","10.111.132.151:6789","10.101.46.100:6789"]}] data:c=10.101.46.100:6789,a=10.100.186.100:6789,b=10.111.132.151:6789 mapping:{"node":{"a":{"Name":"smithi038","Hostname":"smithi038","Address":"172.21.15.38"},"b":{"Name":"smithi195","Hostname":"smithi195","Address":"172.21.15.195"},"c":{"Name":"smithi037","Hostname":"smithi037","Address":"172.21.15.37"}}} maxMonId:2] 2022-03-10T03:15:20.317 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:20.321417 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-provisioner" 2022-03-10T03:15:20.737 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:20.741695 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-node" 2022-03-10T03:15:20.741 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:20.742415 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-10T03:15:20.742 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:20.742641 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-10T03:15:23.118 INFO:teuthology.orchestra.run.smithi037.stderr:+ '[' 11 '!=' 12 ']' 2022-03-10T03:15:23.119 INFO:teuthology.orchestra.run.smithi037.stdout:waiting for osd to come back up 2022-03-10T03:15:23.119 INFO:teuthology.orchestra.run.smithi037.stderr:+ echo 'waiting for osd to come back up' 2022-03-10T03:15:23.120 INFO:teuthology.orchestra.run.smithi037.stderr:++ toolbox ceph osd stat 2022-03-10T03:15:23.120 INFO:teuthology.orchestra.run.smithi037.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-03-10T03:15:23.120 INFO:teuthology.orchestra.run.smithi037.stderr:++ cut -f3 -d ' ' 2022-03-10T03:15:23.201 INFO:teuthology.orchestra.run.smithi037.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-10T03:15:23.540 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:23.544182 I | op-mon: targeting the mon count 3 2022-03-10T03:15:23.546 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:23.551162 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-10T03:15:23.703 INFO:teuthology.orchestra.run.smithi037.stderr:+ num_osd=11 2022-03-10T03:15:23.703 INFO:teuthology.orchestra.run.smithi037.stderr:+ sleep 30s 2022-03-10T03:15:23.896 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:23.900228 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-10T03:15:23.897 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:23.900258 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-03-10T03:15:23.941 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:23.945077 I | ceph-csi: created kubernetes csi secrets for cluster "rook-ceph" 2022-03-10T03:15:23.941 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:23.945111 I | cephclient: getting or creating ceph auth key "client.crash" 2022-03-10T03:15:24.255 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:24.259666 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-03-10T03:15:24.256 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:24.259695 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-10T03:15:24.364 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:24.369153 I | ceph-crashcollector-controller: created kubernetes crash collector secret for cluster "rook-ceph" 2022-03-10T03:15:24.661 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:24.665564 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-10T03:15:24.661 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:24.665591 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-03-10T03:15:24.846 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:24.849793 I | cephclient: successfully enabled msgr2 protocol 2022-03-10T03:15:24.846 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:24.849819 I | cephclient: create rbd-mirror bootstrap peer token "client.rbd-mirror-peer" 2022-03-10T03:15:24.846 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:24.849823 I | cephclient: getting or creating ceph auth key "client.rbd-mirror-peer" 2022-03-10T03:15:25.007 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:25.011192 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-03-10T03:15:25.007 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:25.011219 I | op-config: deleting "log file" option from the mon configuration database 2022-03-10T03:15:25.290 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:25.294196 I | cephclient: successfully created rbd-mirror bootstrap peer token for cluster "rook-ceph" 2022-03-10T03:15:25.325 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:25.329752 I | op-mgr: start running mgr 2022-03-10T03:15:25.326 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:25.329788 I | cephclient: getting or creating ceph auth key "mgr.a" 2022-03-10T03:15:25.358 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:25.362186 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-03-10T03:15:25.358 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:25.362206 I | op-mon: checking for basic quorum with existing mons 2022-03-10T03:15:25.540 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:25.543950 I | op-mon: mon "a" endpoint is [v2:10.100.186.100:3300,v1:10.100.186.100:6789] 2022-03-10T03:15:26.740 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:26.744764 I | op-mon: mon "b" endpoint is [v2:10.111.132.151:3300,v1:10.111.132.151:6789] 2022-03-10T03:15:26.973 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:26.977950 I | op-mgr: deployment for mgr rook-ceph-mgr-a already exists. updating if needed 2022-03-10T03:15:26.986 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:26.990601 I | op-k8sutil: deployment "rook-ceph-mgr-a" did not change, nothing to update 2022-03-10T03:15:26.990 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:26.994618 I | op-mgr: setting services to point to mgr "a" 2022-03-10T03:15:28.360 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:28.364032 I | op-mon: mon "c" endpoint is [v2:10.101.46.100:3300,v1:10.101.46.100:6789] 2022-03-10T03:15:28.939 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:28.942813 I | op-mgr: no need to update service "rook-ceph-mgr" 2022-03-10T03:15:28.939 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:28.942948 I | op-mgr: successful modules: balancer 2022-03-10T03:15:28.956 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:28.959875 I | op-osd: start running osds in namespace "rook-ceph" 2022-03-10T03:15:28.956 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:28.959909 I | op-osd: wait timeout for healthy OSDs during upgrade or restart is "10m0s" 2022-03-10T03:15:29.019 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:29.023145 I | op-osd: start provisioning the OSDs on PVCs, if needed 2022-03-10T03:15:29.343 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:29.345003 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi037-lv-1" 2022-03-10T03:15:29.343 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:29.345039 I | op-osd: OSD PVC "smithi037-lv-1-data-08t564" already exists 2022-03-10T03:15:29.344 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:29.345048 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi037-lv-2" 2022-03-10T03:15:29.344 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:29.345054 I | op-osd: OSD PVC "smithi037-lv-2-data-0hgjtd" already exists 2022-03-10T03:15:29.344 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:29.345060 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi037-lv-3" 2022-03-10T03:15:29.344 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:29.345065 I | op-osd: OSD PVC "smithi037-lv-3-data-0z8qlf" already exists 2022-03-10T03:15:29.345 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:29.345073 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi037-lv-4" 2022-03-10T03:15:29.345 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:29.345079 I | op-osd: OSD PVC "smithi037-lv-4-data-0pfq7x" already exists 2022-03-10T03:15:29.345 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:29.345083 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi038-lv-2" 2022-03-10T03:15:29.345 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:29.345088 I | op-osd: OSD PVC "smithi038-lv-2-data-0hvtxr" already exists 2022-03-10T03:15:29.345 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:29.345093 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi038-lv-3" 2022-03-10T03:15:29.346 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:29.345098 I | op-osd: OSD PVC "smithi038-lv-3-data-0h4lqd" already exists 2022-03-10T03:15:29.346 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:29.345102 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi038-lv-4" 2022-03-10T03:15:29.346 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:29.345108 I | op-osd: OSD PVC "smithi038-lv-4-data-0pzz62" already exists 2022-03-10T03:15:29.346 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:29.345113 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi195-lv-1" 2022-03-10T03:15:29.347 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:29.345119 I | op-osd: OSD PVC "smithi195-lv-1-data-0jjrtk" already exists 2022-03-10T03:15:29.347 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:29.345123 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi195-lv-2" 2022-03-10T03:15:29.347 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:29.345131 I | op-osd: OSD PVC "smithi195-lv-2-data-0cgctr" already exists 2022-03-10T03:15:29.347 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:29.345146 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi195-lv-3" 2022-03-10T03:15:29.347 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:29.345157 I | op-osd: OSD PVC "smithi195-lv-3-data-0qs2g7" already exists 2022-03-10T03:15:29.348 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:29.345164 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi195-lv-4" 2022-03-10T03:15:29.348 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:29.345176 I | op-osd: OSD PVC "smithi195-lv-4-data-0v9ftf" already exists 2022-03-10T03:15:29.348 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:29.345187 I | op-osd: creating 1 new PVCs for device set "smithi038-lv-1" 2022-03-10T03:15:29.540 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:29.544176 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.100.186.100:6789","10.111.132.151:6789","10.101.46.100:6789"]}] data:a=10.100.186.100:6789,b=10.111.132.151:6789,c=10.101.46.100:6789 mapping:{"node":{"a":{"Name":"smithi038","Hostname":"smithi038","Address":"172.21.15.38"},"b":{"Name":"smithi195","Hostname":"smithi195","Address":"172.21.15.195"},"c":{"Name":"smithi037","Hostname":"smithi037","Address":"172.21.15.37"}}} maxMonId:2] 2022-03-10T03:15:29.742 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:29.746560 I | op-osd: successfully provisioned PVC "smithi038-lv-1-data-0rrtwp" 2022-03-10T03:15:29.803 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:29.805998 I | op-osd: OSD will have its main bluestore block on "smithi037-lv-1-data-08t564" 2022-03-10T03:15:29.803 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:29.806023 I | op-osd: skipping OSD prepare job creation for PVC "smithi037-lv-1-data-08t564" because OSD daemon using the PVC already exists 2022-03-10T03:15:29.804 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:29.806027 I | op-osd: OSD will have its main bluestore block on "smithi037-lv-2-data-0hgjtd" 2022-03-10T03:15:29.804 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:29.806030 I | op-osd: skipping OSD prepare job creation for PVC "smithi037-lv-2-data-0hgjtd" because OSD daemon using the PVC already exists 2022-03-10T03:15:29.804 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:29.806034 I | op-osd: OSD will have its main bluestore block on "smithi037-lv-3-data-0z8qlf" 2022-03-10T03:15:29.804 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:29.806037 I | op-osd: skipping OSD prepare job creation for PVC "smithi037-lv-3-data-0z8qlf" because OSD daemon using the PVC already exists 2022-03-10T03:15:29.805 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:29.806040 I | op-osd: OSD will have its main bluestore block on "smithi037-lv-4-data-0pfq7x" 2022-03-10T03:15:29.805 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:29.806043 I | op-osd: skipping OSD prepare job creation for PVC "smithi037-lv-4-data-0pfq7x" because OSD daemon using the PVC already exists 2022-03-10T03:15:29.805 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:29.806046 I | op-osd: OSD will have its main bluestore block on "smithi038-lv-2-data-0hvtxr" 2022-03-10T03:15:29.805 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:29.806049 I | op-osd: skipping OSD prepare job creation for PVC "smithi038-lv-2-data-0hvtxr" because OSD daemon using the PVC already exists 2022-03-10T03:15:29.805 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:29.806052 I | op-osd: OSD will have its main bluestore block on "smithi038-lv-3-data-0h4lqd" 2022-03-10T03:15:29.807 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:29.806055 I | op-osd: skipping OSD prepare job creation for PVC "smithi038-lv-3-data-0h4lqd" because OSD daemon using the PVC already exists 2022-03-10T03:15:29.807 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:29.806058 I | op-osd: OSD will have its main bluestore block on "smithi038-lv-4-data-0pzz62" 2022-03-10T03:15:29.808 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:29.806060 I | op-osd: skipping OSD prepare job creation for PVC "smithi038-lv-4-data-0pzz62" because OSD daemon using the PVC already exists 2022-03-10T03:15:29.808 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:29.806063 I | op-osd: OSD will have its main bluestore block on "smithi195-lv-1-data-0jjrtk" 2022-03-10T03:15:29.808 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:29.806067 I | op-osd: skipping OSD prepare job creation for PVC "smithi195-lv-1-data-0jjrtk" because OSD daemon using the PVC already exists 2022-03-10T03:15:29.808 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:29.806070 I | op-osd: OSD will have its main bluestore block on "smithi195-lv-2-data-0cgctr" 2022-03-10T03:15:29.809 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:29.806074 I | op-osd: skipping OSD prepare job creation for PVC "smithi195-lv-2-data-0cgctr" because OSD daemon using the PVC already exists 2022-03-10T03:15:29.809 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:29.806077 I | op-osd: OSD will have its main bluestore block on "smithi195-lv-3-data-0qs2g7" 2022-03-10T03:15:29.809 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:29.806080 I | op-osd: skipping OSD prepare job creation for PVC "smithi195-lv-3-data-0qs2g7" because OSD daemon using the PVC already exists 2022-03-10T03:15:29.809 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:29.806083 I | op-osd: OSD will have its main bluestore block on "smithi195-lv-4-data-0v9ftf" 2022-03-10T03:15:29.809 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:29.806087 I | op-osd: skipping OSD prepare job creation for PVC "smithi195-lv-4-data-0v9ftf" because OSD daemon using the PVC already exists 2022-03-10T03:15:29.810 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:29.806091 I | op-osd: OSD will have its main bluestore block on "smithi038-lv-1-data-0rrtwp" 2022-03-10T03:15:30.203 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:30.207287 I | op-mgr: successful modules: prometheus 2022-03-10T03:15:30.206 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:30.210657 I | op-mgr: successful modules: mgr module(s) from the spec 2022-03-10T03:15:30.210 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:30.214513 I | op-mgr: successful modules: dashboard 2022-03-10T03:15:30.584 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:30.588964 I | op-osd: started OSD provisioning job for PVC "smithi038-lv-1-data-0rrtwp" 2022-03-10T03:15:30.585 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:30.588990 I | op-osd: start provisioning the OSDs on nodes, if needed 2022-03-10T03:15:30.585 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:30.588996 I | op-osd: no nodes are defined for configuring OSDs on raw devices 2022-03-10T03:15:30.736 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:30.741058 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-10T03:15:30.740 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:30.741349 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-10T03:15:30.940 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:30.943718 I | op-osd: OSD orchestration status for PVC smithi038-lv-1-data-0rrtwp is "starting" 2022-03-10T03:15:31.041 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:31.045375 I | op-osd: skipping osd checks for ok-to-stop 2022-03-10T03:15:31.048 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:31.052902 I | op-osd: updating OSD 1 on PVC "smithi037-lv-3-data-0z8qlf" 2022-03-10T03:15:31.049 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:31.052943 I | op-osd: OSD will have its main bluestore block on "smithi037-lv-3-data-0z8qlf" 2022-03-10T03:15:31.253 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:31.257824 I | op-osd: skipping osd checks for ok-to-stop 2022-03-10T03:15:31.266 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:31.267705 I | op-osd: updating OSD 10 on PVC "smithi195-lv-2-data-0cgctr" 2022-03-10T03:15:31.266 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:31.267778 I | op-osd: OSD will have its main bluestore block on "smithi195-lv-2-data-0cgctr" 2022-03-10T03:15:31.442 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:31.446980 I | op-mon: deployment for mon rook-ceph-mon-a already exists. updating if needed 2022-03-10T03:15:31.451 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:31.455907 I | op-k8sutil: deployment "rook-ceph-mon-a" did not change, nothing to update 2022-03-10T03:15:31.452 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:31.455934 I | op-mon: waiting for mon quorum with [a b c] 2022-03-10T03:15:31.543 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:31.547541 I | op-osd: skipping osd checks for ok-to-stop 2022-03-10T03:15:31.561 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:31.565832 I | op-osd: updating OSD 11 on PVC "smithi037-lv-4-data-0pfq7x" 2022-03-10T03:15:31.562 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:31.565856 I | op-osd: OSD will have its main bluestore block on "smithi037-lv-4-data-0pfq7x" 2022-03-10T03:15:31.817 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:31.821374 I | op-mgr: successful modules: orchestrator modules 2022-03-10T03:15:31.949 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:31.953250 I | op-mon: mons running: [a b c] 2022-03-10T03:15:32.154 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:32.158349 I | op-osd: skipping osd checks for ok-to-stop 2022-03-10T03:15:32.249 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:32.253352 I | op-osd: updating OSD 2 on PVC "smithi037-lv-2-data-0hgjtd" 2022-03-10T03:15:32.249 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:32.253391 I | op-osd: OSD will have its main bluestore block on "smithi037-lv-2-data-0hgjtd" 2022-03-10T03:15:32.367 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:32.371640 I | op-mon: Monitors in quorum: [a b c] 2022-03-10T03:15:32.647 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:32.652095 I | op-mon: deployment for mon rook-ceph-mon-b already exists. updating if needed 2022-03-10T03:15:33.056 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:33.060823 I | op-k8sutil: deployment "rook-ceph-mon-b" did not change, nothing to update 2022-03-10T03:15:33.057 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:33.060865 I | op-mon: waiting for mon quorum with [a b c] 2022-03-10T03:15:33.093 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:33.097740 I | op-mon: mons running: [a b c] 2022-03-10T03:15:33.356 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:33.361095 I | op-osd: skipping osd checks for ok-to-stop 2022-03-10T03:15:33.448 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:33.452415 I | op-osd: updating OSD 3 on PVC "smithi038-lv-4-data-0pzz62" 2022-03-10T03:15:33.448 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:33.452451 I | op-osd: OSD will have its main bluestore block on "smithi038-lv-4-data-0pzz62" 2022-03-10T03:15:33.517 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:33.521876 I | op-mon: Monitors in quorum: [a b c] 2022-03-10T03:15:33.848 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:33.852148 I | op-mon: deployment for mon rook-ceph-mon-c already exists. updating if needed 2022-03-10T03:15:34.254 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:34.258582 I | op-k8sutil: deployment "rook-ceph-mon-c" did not change, nothing to update 2022-03-10T03:15:34.255 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:34.258614 I | op-mon: waiting for mon quorum with [a b c] 2022-03-10T03:15:34.293 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:34.297216 I | op-mon: mons running: [a b c] 2022-03-10T03:15:34.553 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:34.557642 I | op-osd: skipping osd checks for ok-to-stop 2022-03-10T03:15:34.651 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:34.653132 I | op-osd: updating OSD 4 on PVC "smithi038-lv-3-data-0h4lqd" 2022-03-10T03:15:34.651 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:34.653179 I | op-osd: OSD will have its main bluestore block on "smithi038-lv-3-data-0h4lqd" 2022-03-10T03:15:34.741 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:34.744506 I | op-mon: Monitors in quorum: [a b c] 2022-03-10T03:15:34.742 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:34.744528 I | op-mon: mons created: 3 2022-03-10T03:15:35.182 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:35.186683 I | op-mon: waiting for mon quorum with [a b c] 2022-03-10T03:15:35.218 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:35.222576 I | op-mon: mons running: [a b c] 2022-03-10T03:15:35.353 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:35.357873 I | op-osd: skipping osd checks for ok-to-stop 2022-03-10T03:15:35.453 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:35.452970 I | op-osd: updating OSD 5 on PVC "smithi038-lv-2-data-0hvtxr" 2022-03-10T03:15:35.453 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:35.453012 I | op-osd: OSD will have its main bluestore block on "smithi038-lv-2-data-0hvtxr" 2022-03-10T03:15:35.634 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:35.637893 I | op-mon: Monitors in quorum: [a b c] 2022-03-10T03:15:35.634 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:35.637936 I | cephclient: getting or creating ceph auth key "client.csi-rbd-provisioner" 2022-03-10T03:15:36.073 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:36.077336 I | cephclient: getting or creating ceph auth key "client.csi-rbd-node" 2022-03-10T03:15:36.156 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:36.160006 I | op-osd: skipping osd checks for ok-to-stop 2022-03-10T03:15:36.269 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:36.273349 I | op-osd: updating OSD 6 on PVC "smithi195-lv-3-data-0qs2g7" 2022-03-10T03:15:36.269 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:36.273392 I | op-osd: OSD will have its main bluestore block on "smithi195-lv-3-data-0qs2g7" 2022-03-10T03:15:36.505 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:36.509883 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-provisioner" 2022-03-10T03:15:36.958 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:36.962533 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-node" 2022-03-10T03:15:36.967 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:36.971938 I | op-osd: skipping osd checks for ok-to-stop 2022-03-10T03:15:37.047 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:37.051926 I | op-osd: updating OSD 7 on PVC "smithi195-lv-1-data-0jjrtk" 2022-03-10T03:15:37.048 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:37.051960 I | op-osd: OSD will have its main bluestore block on "smithi195-lv-1-data-0jjrtk" 2022-03-10T03:15:37.475 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:37.479768 I | ceph-csi: created kubernetes csi secrets for cluster "rook-ceph" 2022-03-10T03:15:37.476 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:37.479801 I | cephclient: getting or creating ceph auth key "client.crash" 2022-03-10T03:15:37.755 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:37.759404 I | op-osd: skipping osd checks for ok-to-stop 2022-03-10T03:15:37.848 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:37.851958 I | op-osd: updating OSD 8 on PVC "smithi037-lv-1-data-08t564" 2022-03-10T03:15:37.848 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:37.851974 I | op-osd: OSD will have its main bluestore block on "smithi037-lv-1-data-08t564" 2022-03-10T03:15:37.910 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:37.914623 I | ceph-crashcollector-controller: created kubernetes crash collector secret for cluster "rook-ceph" 2022-03-10T03:15:38.461 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:38.465309 I | cephclient: successfully enabled msgr2 protocol 2022-03-10T03:15:38.461 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:38.465347 I | cephclient: create rbd-mirror bootstrap peer token "client.rbd-mirror-peer" 2022-03-10T03:15:38.462 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:38.465357 I | cephclient: getting or creating ceph auth key "client.rbd-mirror-peer" 2022-03-10T03:15:38.556 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:38.560994 I | op-osd: skipping osd checks for ok-to-stop 2022-03-10T03:15:38.648 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:38.653053 I | op-osd: updating OSD 9 on PVC "smithi195-lv-4-data-0v9ftf" 2022-03-10T03:15:38.649 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:38.653077 I | op-osd: OSD will have its main bluestore block on "smithi195-lv-4-data-0v9ftf" 2022-03-10T03:15:39.128 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:39.131910 I | cephclient: successfully created rbd-mirror bootstrap peer token for cluster "rook-ceph" 2022-03-10T03:15:39.156 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:39.160250 I | op-mgr: start running mgr 2022-03-10T03:15:39.156 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:39.160306 I | cephclient: getting or creating ceph auth key "mgr.a" 2022-03-10T03:15:39.675 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:39.680017 I | op-mgr: deployment for mgr rook-ceph-mgr-a already exists. updating if needed 2022-03-10T03:15:39.683 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:39.688036 I | op-k8sutil: deployment "rook-ceph-mgr-a" did not change, nothing to update 2022-03-10T03:15:39.848 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:39.852015 I | op-mgr: setting services to point to mgr "a" 2022-03-10T03:15:39.987 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:39.992041 I | op-mgr: no need to update service "rook-ceph-mgr" 2022-03-10T03:15:39.988 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:39.992141 I | op-mgr: successful modules: balancer 2022-03-10T03:15:40.000 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:40.004401 I | op-osd: start running osds in namespace "rook-ceph" 2022-03-10T03:15:40.000 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:40.004429 I | op-osd: wait timeout for healthy OSDs during upgrade or restart is "10m0s" 2022-03-10T03:15:40.086 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:40.090275 I | op-osd: start provisioning the OSDs on PVCs, if needed 2022-03-10T03:15:40.093 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:40.095362 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi037-lv-1" 2022-03-10T03:15:40.093 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:40.095394 I | op-osd: OSD PVC "smithi037-lv-1-data-08t564" already exists 2022-03-10T03:15:40.093 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:40.095405 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi037-lv-2" 2022-03-10T03:15:40.093 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:40.095415 I | op-osd: OSD PVC "smithi037-lv-2-data-0hgjtd" already exists 2022-03-10T03:15:40.094 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:40.095422 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi037-lv-3" 2022-03-10T03:15:40.094 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:40.095432 I | op-osd: OSD PVC "smithi037-lv-3-data-0z8qlf" already exists 2022-03-10T03:15:40.095 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:40.095441 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi037-lv-4" 2022-03-10T03:15:40.095 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:40.095451 I | op-osd: OSD PVC "smithi037-lv-4-data-0pfq7x" already exists 2022-03-10T03:15:40.095 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:40.095459 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi038-lv-2" 2022-03-10T03:15:40.096 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:40.095469 I | op-osd: OSD PVC "smithi038-lv-2-data-0hvtxr" already exists 2022-03-10T03:15:40.096 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:40.095477 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi038-lv-3" 2022-03-10T03:15:40.096 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:40.095487 I | op-osd: OSD PVC "smithi038-lv-3-data-0h4lqd" already exists 2022-03-10T03:15:40.096 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:40.095493 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi038-lv-4" 2022-03-10T03:15:40.097 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:40.095505 I | op-osd: OSD PVC "smithi038-lv-4-data-0pzz62" already exists 2022-03-10T03:15:40.097 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:40.095514 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi195-lv-1" 2022-03-10T03:15:40.097 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:40.095524 I | op-osd: OSD PVC "smithi195-lv-1-data-0jjrtk" already exists 2022-03-10T03:15:40.098 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:40.095531 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi195-lv-2" 2022-03-10T03:15:40.098 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:40.095540 I | op-osd: OSD PVC "smithi195-lv-2-data-0cgctr" already exists 2022-03-10T03:15:40.098 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:40.095549 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi195-lv-3" 2022-03-10T03:15:40.098 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:40.095559 I | op-osd: OSD PVC "smithi195-lv-3-data-0qs2g7" already exists 2022-03-10T03:15:40.099 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:40.095566 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi195-lv-4" 2022-03-10T03:15:40.099 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:40.095574 I | op-osd: OSD PVC "smithi195-lv-4-data-0v9ftf" already exists 2022-03-10T03:15:40.099 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:40.095582 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi038-lv-1" 2022-03-10T03:15:40.099 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:40.095594 I | op-osd: OSD PVC "smithi038-lv-1-data-0rrtwp" already exists 2022-03-10T03:15:40.270 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:40.272719 I | op-osd: OSD will have its main bluestore block on "smithi037-lv-1-data-08t564" 2022-03-10T03:15:40.271 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:40.272739 I | op-osd: skipping OSD prepare job creation for PVC "smithi037-lv-1-data-08t564" because OSD daemon using the PVC already exists 2022-03-10T03:15:40.271 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:40.272744 I | op-osd: OSD will have its main bluestore block on "smithi037-lv-2-data-0hgjtd" 2022-03-10T03:15:40.271 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:40.272748 I | op-osd: skipping OSD prepare job creation for PVC "smithi037-lv-2-data-0hgjtd" because OSD daemon using the PVC already exists 2022-03-10T03:15:40.271 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:40.272753 I | op-osd: OSD will have its main bluestore block on "smithi037-lv-3-data-0z8qlf" 2022-03-10T03:15:40.272 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:40.272757 I | op-osd: skipping OSD prepare job creation for PVC "smithi037-lv-3-data-0z8qlf" because OSD daemon using the PVC already exists 2022-03-10T03:15:40.272 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:40.272761 I | op-osd: OSD will have its main bluestore block on "smithi037-lv-4-data-0pfq7x" 2022-03-10T03:15:40.272 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:40.272765 I | op-osd: skipping OSD prepare job creation for PVC "smithi037-lv-4-data-0pfq7x" because OSD daemon using the PVC already exists 2022-03-10T03:15:40.272 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:40.272771 I | op-osd: OSD will have its main bluestore block on "smithi038-lv-2-data-0hvtxr" 2022-03-10T03:15:40.273 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:40.272775 I | op-osd: skipping OSD prepare job creation for PVC "smithi038-lv-2-data-0hvtxr" because OSD daemon using the PVC already exists 2022-03-10T03:15:40.273 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:40.272782 I | op-osd: OSD will have its main bluestore block on "smithi038-lv-3-data-0h4lqd" 2022-03-10T03:15:40.273 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:40.272789 I | op-osd: skipping OSD prepare job creation for PVC "smithi038-lv-3-data-0h4lqd" because OSD daemon using the PVC already exists 2022-03-10T03:15:40.273 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:40.272795 I | op-osd: OSD will have its main bluestore block on "smithi038-lv-4-data-0pzz62" 2022-03-10T03:15:40.273 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:40.272800 I | op-osd: skipping OSD prepare job creation for PVC "smithi038-lv-4-data-0pzz62" because OSD daemon using the PVC already exists 2022-03-10T03:15:40.274 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:40.272805 I | op-osd: OSD will have its main bluestore block on "smithi195-lv-1-data-0jjrtk" 2022-03-10T03:15:40.274 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:40.272810 I | op-osd: skipping OSD prepare job creation for PVC "smithi195-lv-1-data-0jjrtk" because OSD daemon using the PVC already exists 2022-03-10T03:15:40.274 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:40.272815 I | op-osd: OSD will have its main bluestore block on "smithi195-lv-2-data-0cgctr" 2022-03-10T03:15:40.274 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:40.272820 I | op-osd: skipping OSD prepare job creation for PVC "smithi195-lv-2-data-0cgctr" because OSD daemon using the PVC already exists 2022-03-10T03:15:40.275 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:40.272826 I | op-osd: OSD will have its main bluestore block on "smithi195-lv-3-data-0qs2g7" 2022-03-10T03:15:40.275 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:40.272831 I | op-osd: skipping OSD prepare job creation for PVC "smithi195-lv-3-data-0qs2g7" because OSD daemon using the PVC already exists 2022-03-10T03:15:40.275 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:40.272836 I | op-osd: OSD will have its main bluestore block on "smithi195-lv-4-data-0v9ftf" 2022-03-10T03:15:40.275 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:40.272841 I | op-osd: skipping OSD prepare job creation for PVC "smithi195-lv-4-data-0v9ftf" because OSD daemon using the PVC already exists 2022-03-10T03:15:40.275 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:40.272847 I | op-osd: OSD will have its main bluestore block on "smithi038-lv-1-data-0rrtwp" 2022-03-10T03:15:40.279 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:40.283581 I | op-k8sutil: Found previous job rook-ceph-osd-prepare-smithi038-lv-1-data-0rrtwp. Status={Conditions:[] StartTime:2022-03-10 03:15:30 +0000 UTC CompletionTime: Active:1 Succeeded:0 Failed:0 CompletedIndexes: UncountedTerminatedPods:&UncountedTerminatedPods{Succeeded:[],Failed:[],}} 2022-03-10T03:15:40.279 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:40.283596 I | op-osd: started OSD provisioning job for PVC "smithi038-lv-1-data-0rrtwp" 2022-03-10T03:15:40.279 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:40.283602 I | op-osd: start provisioning the OSDs on nodes, if needed 2022-03-10T03:15:40.280 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:40.283605 I | op-osd: no nodes are defined for configuring OSDs on raw devices 2022-03-10T03:15:40.281 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:40.285984 I | op-osd: OSD orchestration status for PVC smithi038-lv-1-data-0rrtwp is "starting" 2022-03-10T03:15:40.385 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:40.387543 I | op-osd: skipping osd checks for ok-to-stop 2022-03-10T03:15:40.448 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:40.452950 I | op-osd: updating OSD 1 on PVC "smithi037-lv-3-data-0z8qlf" 2022-03-10T03:15:40.449 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:40.452983 I | op-osd: OSD will have its main bluestore block on "smithi037-lv-3-data-0z8qlf" 2022-03-10T03:15:41.027 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:41.031026 I | op-mgr: successful modules: dashboard 2022-03-10T03:15:41.029 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:41.033722 I | op-mgr: successful modules: prometheus 2022-03-10T03:15:41.048 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:41.052660 I | op-mgr: successful modules: mgr module(s) from the spec 2022-03-10T03:15:41.151 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:41.155708 I | op-osd: skipping osd checks for ok-to-stop 2022-03-10T03:15:41.245 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:41.249877 I | op-osd: updating OSD 10 on PVC "smithi195-lv-2-data-0cgctr" 2022-03-10T03:15:41.246 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:41.249897 I | op-osd: OSD will have its main bluestore block on "smithi195-lv-2-data-0cgctr" 2022-03-10T03:15:41.964 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:41.968099 I | op-osd: skipping osd checks for ok-to-stop 2022-03-10T03:15:42.069 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:42.073699 I | op-osd: updating OSD 11 on PVC "smithi037-lv-4-data-0pfq7x" 2022-03-10T03:15:42.069 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:42.073737 I | op-osd: OSD will have its main bluestore block on "smithi037-lv-4-data-0pfq7x" 2022-03-10T03:15:42.756 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:42.760837 I | op-osd: skipping osd checks for ok-to-stop 2022-03-10T03:15:42.849 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:42.853703 I | op-osd: updating OSD 2 on PVC "smithi037-lv-2-data-0hgjtd" 2022-03-10T03:15:42.850 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:42.853740 I | op-osd: OSD will have its main bluestore block on "smithi037-lv-2-data-0hgjtd" 2022-03-10T03:15:43.085 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:43.089920 I | op-mgr: successful modules: orchestrator modules 2022-03-10T03:15:43.550 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:43.554859 I | op-osd: skipping osd checks for ok-to-stop 2022-03-10T03:15:43.649 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:43.653104 I | op-osd: updating OSD 3 on PVC "smithi038-lv-4-data-0pzz62" 2022-03-10T03:15:43.649 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:43.653136 I | op-osd: OSD will have its main bluestore block on "smithi038-lv-4-data-0pzz62" 2022-03-10T03:15:44.352 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:44.356686 I | op-osd: skipping osd checks for ok-to-stop 2022-03-10T03:15:44.449 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:44.453542 I | op-osd: updating OSD 4 on PVC "smithi038-lv-3-data-0h4lqd" 2022-03-10T03:15:44.449 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:44.453585 I | op-osd: OSD will have its main bluestore block on "smithi038-lv-3-data-0h4lqd" 2022-03-10T03:15:45.171 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:45.174740 I | op-osd: skipping osd checks for ok-to-stop 2022-03-10T03:15:45.249 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:45.252778 I | op-osd: updating OSD 5 on PVC "smithi038-lv-2-data-0hvtxr" 2022-03-10T03:15:45.250 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:45.252806 I | op-osd: OSD will have its main bluestore block on "smithi038-lv-2-data-0hvtxr" 2022-03-10T03:15:45.292 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:45.296990 I | op-osd: OSD orchestration status for node smithi038-lv-1-data-0rrtwp is "orchestrating" 2022-03-10T03:15:45.686 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:45.689917 I | op-osd: OSD orchestration status for PVC smithi038-lv-1-data-0rrtwp is "orchestrating" 2022-03-10T03:15:45.853 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:45.857557 I | op-osd: OSD orchestration status for node smithi038-lv-1-data-0rrtwp is "orchestrating" 2022-03-10T03:15:45.857 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:45.857859 I | op-osd: OSD orchestration status for PVC smithi038-lv-1-data-0rrtwp is "orchestrating" 2022-03-10T03:15:45.954 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:45.958694 I | op-osd: skipping osd checks for ok-to-stop 2022-03-10T03:15:46.049 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:46.052987 I | op-osd: updating OSD 6 on PVC "smithi195-lv-3-data-0qs2g7" 2022-03-10T03:15:46.049 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:46.053027 I | op-osd: OSD will have its main bluestore block on "smithi195-lv-3-data-0qs2g7" 2022-03-10T03:15:46.756 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:46.760779 I | op-osd: skipping osd checks for ok-to-stop 2022-03-10T03:15:46.849 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:46.853351 I | op-osd: updating OSD 7 on PVC "smithi195-lv-1-data-0jjrtk" 2022-03-10T03:15:46.849 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:46.853393 I | op-osd: OSD will have its main bluestore block on "smithi195-lv-1-data-0jjrtk" 2022-03-10T03:15:47.556 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:47.560232 I | op-osd: skipping osd checks for ok-to-stop 2022-03-10T03:15:47.648 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:47.652243 I | op-osd: updating OSD 8 on PVC "smithi037-lv-1-data-08t564" 2022-03-10T03:15:47.648 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:47.652279 I | op-osd: OSD will have its main bluestore block on "smithi037-lv-1-data-08t564" 2022-03-10T03:15:48.400 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:48.404097 I | op-osd: skipping osd checks for ok-to-stop 2022-03-10T03:15:48.455 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:48.456115 I | op-osd: updating OSD 9 on PVC "smithi195-lv-4-data-0v9ftf" 2022-03-10T03:15:48.455 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:48.456158 I | op-osd: OSD will have its main bluestore block on "smithi195-lv-4-data-0v9ftf" 2022-03-10T03:15:51.846 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:51.850362 I | op-osd: OSD orchestration status for PVC smithi038-lv-1-data-0rrtwp is "completed" 2022-03-10T03:15:51.847 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:51.850396 I | op-osd: creating OSD 0 on PVC "smithi038-lv-1-data-0rrtwp" 2022-03-10T03:15:51.848 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:51.850411 I | op-osd: OSD will have its main bluestore block on "smithi038-lv-1-data-0rrtwp" 2022-03-10T03:15:51.848 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:51.850541 I | op-osd: OSD orchestration status for PVC smithi038-lv-1-data-0rrtwp is "completed" 2022-03-10T03:15:51.848 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:51.850585 I | op-osd: creating OSD 0 on PVC "smithi038-lv-1-data-0rrtwp" 2022-03-10T03:15:51.849 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:51.850609 I | op-osd: OSD will have its main bluestore block on "smithi038-lv-1-data-0rrtwp" 2022-03-10T03:15:51.920 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:51.924736 E | ceph-spec: failed to update cluster condition to {Type:Progressing Status:True Reason:ClusterProgressing Message:Processing OSD 0 on PVC "smithi038-lv-1-data-0rrtwp" LastHeartbeatTime:2022-03-10 03:15:51.874811737 +0000 UTC m=+527.568645819 LastTransitionTime:2022-03-10 03:15:51.874811524 +0000 UTC m=+527.568645633}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-10T03:15:51.967 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:51.971401 E | op-osd: failed to create OSD 0 on PVC "smithi038-lv-1-data-0rrtwp": failed to create deployment for OSD 0 on PVC "smithi038-lv-1-data-0rrtwp": deployments.apps "rook-ceph-osd-0" already exists 2022-03-10T03:15:52.062 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:52.066299 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". failed to reconcile cluster "rook-ceph": failed to configure local ceph cluster: failed to create cluster: failed to start ceph osds: 1 failures encountered while running osds on nodes in namespace "rook-ceph". 2022-03-10T03:15:52.063 INFO:tasks.rook.operator.smithi037.stdout:failed to create OSD 0 on PVC "smithi038-lv-1-data-0rrtwp": failed to create deployment for OSD 0 on PVC "smithi038-lv-1-data-0rrtwp": deployments.apps "rook-ceph-osd-0" already exists 2022-03-10T03:15:52.067 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:52.072043 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-03-10T03:15:52.111 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:52.114864 I | op-mon: parsing mon endpoints: a=10.100.186.100:6789,b=10.111.132.151:6789,c=10.101.46.100:6789 2022-03-10T03:15:52.144 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:52.148358 I | ceph-spec: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3... 2022-03-10T03:15:52.452 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:52.456856 I | op-osd: finished running OSDs in namespace "rook-ceph" 2022-03-10T03:15:52.453 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:52.456885 I | ceph-cluster-controller: done reconciling ceph cluster in namespace "rook-ceph" 2022-03-10T03:15:53.704 INFO:teuthology.orchestra.run.smithi037.stderr:+ '[' 11 '!=' 12 ']' 2022-03-10T03:15:53.705 INFO:teuthology.orchestra.run.smithi037.stdout:waiting for osd to come back up 2022-03-10T03:15:53.706 INFO:teuthology.orchestra.run.smithi037.stderr:+ echo 'waiting for osd to come back up' 2022-03-10T03:15:53.706 INFO:teuthology.orchestra.run.smithi037.stderr:++ toolbox ceph osd stat 2022-03-10T03:15:53.706 INFO:teuthology.orchestra.run.smithi037.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-03-10T03:15:53.707 INFO:teuthology.orchestra.run.smithi037.stderr:++ cut -f3 -d ' ' 2022-03-10T03:15:53.792 INFO:teuthology.orchestra.run.smithi037.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-10T03:15:54.368 INFO:teuthology.orchestra.run.smithi037.stderr:+ num_osd=11 2022-03-10T03:15:54.369 INFO:teuthology.orchestra.run.smithi037.stderr:+ sleep 30s 2022-03-10T03:15:57.902 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:57.906560 I | ceph-spec: detected ceph image version: "17.0.0-11025 quincy" 2022-03-10T03:15:57.903 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:57.906592 I | ceph-cluster-controller: validating ceph version from provided image 2022-03-10T03:15:57.903 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:57.906605 W | ceph-cluster-controller: unsupported ceph version detected: "17.0.0-11025 quincy", pursuing 2022-03-10T03:15:57.961 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:57.965710 I | op-mon: parsing mon endpoints: a=10.100.186.100:6789,b=10.111.132.151:6789,c=10.101.46.100:6789 2022-03-10T03:15:57.969 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:57.974165 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-10T03:15:57.974 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:57.976740 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-10T03:15:58.407 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:58.411259 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-11025 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3" 2022-03-10T03:15:58.574 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:58.578207 I | op-mon: start running mons 2022-03-10T03:15:58.579 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:58.582893 I | op-mon: parsing mon endpoints: a=10.100.186.100:6789,b=10.111.132.151:6789,c=10.101.46.100:6789 2022-03-10T03:15:58.611 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:58.615372 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.100.186.100:6789","10.111.132.151:6789","10.101.46.100:6789"]}] data:a=10.100.186.100:6789,b=10.111.132.151:6789,c=10.101.46.100:6789 mapping:{"node":{"a":{"Name":"smithi038","Hostname":"smithi038","Address":"172.21.15.38"},"b":{"Name":"smithi195","Hostname":"smithi195","Address":"172.21.15.195"},"c":{"Name":"smithi037","Hostname":"smithi037","Address":"172.21.15.37"}}} maxMonId:2] 2022-03-10T03:15:59.039 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:59.043046 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-10T03:15:59.042 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:15:59.043481 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-10T03:16:00.617 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:00.620972 I | op-mon: targeting the mon count 3 2022-03-10T03:16:00.653 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:00.657442 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-10T03:16:01.008 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:01.012103 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-10T03:16:01.009 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:01.012140 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-03-10T03:16:01.461 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:01.465232 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-03-10T03:16:01.462 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:01.465265 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-10T03:16:01.826 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:01.830393 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-10T03:16:01.826 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:01.830413 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-03-10T03:16:02.176 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:02.179934 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-03-10T03:16:02.176 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:02.179964 I | op-config: deleting "log file" option from the mon configuration database 2022-03-10T03:16:02.532 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:02.536076 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-03-10T03:16:02.532 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:02.536113 I | op-mon: checking for basic quorum with existing mons 2022-03-10T03:16:02.674 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:02.678794 I | op-mon: mon "a" endpoint is [v2:10.100.186.100:3300,v1:10.100.186.100:6789] 2022-03-10T03:16:02.827 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:02.831718 I | op-mon: mon "b" endpoint is [v2:10.111.132.151:3300,v1:10.111.132.151:6789] 2022-03-10T03:16:03.018 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:03.022207 I | op-mon: mon "c" endpoint is [v2:10.101.46.100:3300,v1:10.101.46.100:6789] 2022-03-10T03:16:03.617 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:03.621838 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.100.186.100:6789","10.111.132.151:6789","10.101.46.100:6789"]}] data:a=10.100.186.100:6789,b=10.111.132.151:6789,c=10.101.46.100:6789 mapping:{"node":{"a":{"Name":"smithi038","Hostname":"smithi038","Address":"172.21.15.38"},"b":{"Name":"smithi195","Hostname":"smithi195","Address":"172.21.15.195"},"c":{"Name":"smithi037","Hostname":"smithi037","Address":"172.21.15.37"}}} maxMonId:2] 2022-03-10T03:16:04.218 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:04.222249 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-10T03:16:04.218 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:04.222509 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-10T03:16:04.636 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:04.640962 I | op-mon: deployment for mon rook-ceph-mon-a already exists. updating if needed 2022-03-10T03:16:04.650 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:04.654239 I | op-k8sutil: deployment "rook-ceph-mon-a" did not change, nothing to update 2022-03-10T03:16:04.651 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:04.654293 I | op-mon: waiting for mon quorum with [a b c] 2022-03-10T03:16:05.228 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:05.232994 I | op-mon: mons running: [a b c] 2022-03-10T03:16:05.649 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:05.653053 I | op-mon: Monitors in quorum: [a b c] 2022-03-10T03:16:05.654 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:05.659221 I | op-mon: deployment for mon rook-ceph-mon-b already exists. updating if needed 2022-03-10T03:16:05.666 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:05.670958 I | op-k8sutil: deployment "rook-ceph-mon-b" did not change, nothing to update 2022-03-10T03:16:05.667 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:05.670988 I | op-mon: waiting for mon quorum with [a b c] 2022-03-10T03:16:05.828 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:05.833020 I | op-mon: mons running: [a b c] 2022-03-10T03:16:06.256 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:06.260149 I | op-mon: Monitors in quorum: [a b c] 2022-03-10T03:16:06.262 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:06.266460 I | op-mon: deployment for mon rook-ceph-mon-c already exists. updating if needed 2022-03-10T03:16:06.272 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:06.276988 I | op-k8sutil: deployment "rook-ceph-mon-c" did not change, nothing to update 2022-03-10T03:16:06.273 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:06.277014 I | op-mon: waiting for mon quorum with [a b c] 2022-03-10T03:16:06.439 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:06.443475 I | op-mon: mons running: [a b c] 2022-03-10T03:16:06.981 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:06.984969 I | op-mon: Monitors in quorum: [a b c] 2022-03-10T03:16:06.981 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:06.985002 I | op-mon: mons created: 3 2022-03-10T03:16:07.404 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:07.408051 I | op-mon: waiting for mon quorum with [a b c] 2022-03-10T03:16:07.440 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:07.444419 I | op-mon: mons running: [a b c] 2022-03-10T03:16:07.861 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:07.865233 I | op-mon: Monitors in quorum: [a b c] 2022-03-10T03:16:07.861 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:07.865280 I | cephclient: getting or creating ceph auth key "client.csi-rbd-provisioner" 2022-03-10T03:16:08.305 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:08.309202 I | cephclient: getting or creating ceph auth key "client.csi-rbd-node" 2022-03-10T03:16:08.730 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:08.734299 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-provisioner" 2022-03-10T03:16:09.187 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:09.191569 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-node" 2022-03-10T03:16:09.665 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:09.669740 I | ceph-csi: created kubernetes csi secrets for cluster "rook-ceph" 2022-03-10T03:16:09.666 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:09.669769 I | cephclient: getting or creating ceph auth key "client.crash" 2022-03-10T03:16:10.101 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:10.105494 I | ceph-crashcollector-controller: created kubernetes crash collector secret for cluster "rook-ceph" 2022-03-10T03:16:10.534 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:10.537796 I | cephclient: successfully enabled msgr2 protocol 2022-03-10T03:16:10.534 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:10.537827 I | cephclient: create rbd-mirror bootstrap peer token "client.rbd-mirror-peer" 2022-03-10T03:16:10.534 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:10.537835 I | cephclient: getting or creating ceph auth key "client.rbd-mirror-peer" 2022-03-10T03:16:10.959 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:10.963682 I | cephclient: successfully created rbd-mirror bootstrap peer token for cluster "rook-ceph" 2022-03-10T03:16:10.998 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:11.002533 I | op-mgr: start running mgr 2022-03-10T03:16:10.999 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:11.002571 I | cephclient: getting or creating ceph auth key "mgr.a" 2022-03-10T03:16:11.483 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:11.487865 I | op-mgr: deployment for mgr rook-ceph-mgr-a already exists. updating if needed 2022-03-10T03:16:11.493 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:11.497651 I | op-k8sutil: deployment "rook-ceph-mgr-a" did not change, nothing to update 2022-03-10T03:16:11.496 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:11.501040 I | op-mgr: setting services to point to mgr "a" 2022-03-10T03:16:11.657 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:11.661358 I | op-mgr: no need to update service "rook-ceph-mgr" 2022-03-10T03:16:11.658 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:11.661496 I | op-mgr: successful modules: balancer 2022-03-10T03:16:11.669 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:11.673731 I | op-osd: start running osds in namespace "rook-ceph" 2022-03-10T03:16:11.670 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:11.673768 I | op-osd: wait timeout for healthy OSDs during upgrade or restart is "10m0s" 2022-03-10T03:16:11.761 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:11.766059 I | op-osd: start provisioning the OSDs on PVCs, if needed 2022-03-10T03:16:11.866 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:11.868877 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi037-lv-1" 2022-03-10T03:16:11.866 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:11.868932 I | op-osd: OSD PVC "smithi037-lv-1-data-08t564" already exists 2022-03-10T03:16:11.866 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:11.868959 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi037-lv-2" 2022-03-10T03:16:11.867 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:11.868977 I | op-osd: OSD PVC "smithi037-lv-2-data-0hgjtd" already exists 2022-03-10T03:16:11.867 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:11.868993 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi037-lv-3" 2022-03-10T03:16:11.867 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:11.869009 I | op-osd: OSD PVC "smithi037-lv-3-data-0z8qlf" already exists 2022-03-10T03:16:11.867 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:11.869028 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi037-lv-4" 2022-03-10T03:16:11.868 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:11.869044 I | op-osd: OSD PVC "smithi037-lv-4-data-0pfq7x" already exists 2022-03-10T03:16:11.868 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:11.869059 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi038-lv-2" 2022-03-10T03:16:11.868 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:11.869076 I | op-osd: OSD PVC "smithi038-lv-2-data-0hvtxr" already exists 2022-03-10T03:16:11.868 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:11.869091 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi038-lv-3" 2022-03-10T03:16:11.868 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:11.869106 I | op-osd: OSD PVC "smithi038-lv-3-data-0h4lqd" already exists 2022-03-10T03:16:11.869 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:11.869116 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi038-lv-4" 2022-03-10T03:16:11.869 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:11.869137 I | op-osd: OSD PVC "smithi038-lv-4-data-0pzz62" already exists 2022-03-10T03:16:11.869 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:11.869154 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi195-lv-1" 2022-03-10T03:16:11.869 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:11.869177 I | op-osd: OSD PVC "smithi195-lv-1-data-0jjrtk" already exists 2022-03-10T03:16:11.870 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:11.869191 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi195-lv-2" 2022-03-10T03:16:11.870 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:11.869206 I | op-osd: OSD PVC "smithi195-lv-2-data-0cgctr" already exists 2022-03-10T03:16:11.870 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:11.869222 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi195-lv-3" 2022-03-10T03:16:11.870 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:11.869238 I | op-osd: OSD PVC "smithi195-lv-3-data-0qs2g7" already exists 2022-03-10T03:16:11.870 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:11.869253 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi195-lv-4" 2022-03-10T03:16:11.871 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:11.869269 I | op-osd: OSD PVC "smithi195-lv-4-data-0v9ftf" already exists 2022-03-10T03:16:11.871 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:11.869285 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi038-lv-1" 2022-03-10T03:16:11.871 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:11.869300 I | op-osd: OSD PVC "smithi038-lv-1-data-0rrtwp" already exists 2022-03-10T03:16:11.894 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:11.896002 I | op-osd: OSD will have its main bluestore block on "smithi037-lv-1-data-08t564" 2022-03-10T03:16:11.894 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:11.896025 I | op-osd: skipping OSD prepare job creation for PVC "smithi037-lv-1-data-08t564" because OSD daemon using the PVC already exists 2022-03-10T03:16:11.894 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:11.896029 I | op-osd: OSD will have its main bluestore block on "smithi037-lv-2-data-0hgjtd" 2022-03-10T03:16:11.895 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:11.896032 I | op-osd: skipping OSD prepare job creation for PVC "smithi037-lv-2-data-0hgjtd" because OSD daemon using the PVC already exists 2022-03-10T03:16:11.895 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:11.896036 I | op-osd: OSD will have its main bluestore block on "smithi037-lv-3-data-0z8qlf" 2022-03-10T03:16:11.902 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:11.896039 I | op-osd: skipping OSD prepare job creation for PVC "smithi037-lv-3-data-0z8qlf" because OSD daemon using the PVC already exists 2022-03-10T03:16:11.903 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:11.896042 I | op-osd: OSD will have its main bluestore block on "smithi037-lv-4-data-0pfq7x" 2022-03-10T03:16:11.903 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:11.896045 I | op-osd: skipping OSD prepare job creation for PVC "smithi037-lv-4-data-0pfq7x" because OSD daemon using the PVC already exists 2022-03-10T03:16:11.903 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:11.896048 I | op-osd: OSD will have its main bluestore block on "smithi038-lv-2-data-0hvtxr" 2022-03-10T03:16:11.903 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:11.896051 I | op-osd: skipping OSD prepare job creation for PVC "smithi038-lv-2-data-0hvtxr" because OSD daemon using the PVC already exists 2022-03-10T03:16:11.903 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:11.896054 I | op-osd: OSD will have its main bluestore block on "smithi038-lv-3-data-0h4lqd" 2022-03-10T03:16:11.904 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:11.896057 I | op-osd: skipping OSD prepare job creation for PVC "smithi038-lv-3-data-0h4lqd" because OSD daemon using the PVC already exists 2022-03-10T03:16:11.904 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:11.896060 I | op-osd: OSD will have its main bluestore block on "smithi038-lv-4-data-0pzz62" 2022-03-10T03:16:11.904 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:11.896063 I | op-osd: skipping OSD prepare job creation for PVC "smithi038-lv-4-data-0pzz62" because OSD daemon using the PVC already exists 2022-03-10T03:16:11.904 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:11.896066 I | op-osd: OSD will have its main bluestore block on "smithi195-lv-1-data-0jjrtk" 2022-03-10T03:16:11.905 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:11.896069 I | op-osd: skipping OSD prepare job creation for PVC "smithi195-lv-1-data-0jjrtk" because OSD daemon using the PVC already exists 2022-03-10T03:16:11.905 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:11.896072 I | op-osd: OSD will have its main bluestore block on "smithi195-lv-2-data-0cgctr" 2022-03-10T03:16:11.905 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:11.896074 I | op-osd: skipping OSD prepare job creation for PVC "smithi195-lv-2-data-0cgctr" because OSD daemon using the PVC already exists 2022-03-10T03:16:11.905 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:11.896079 I | op-osd: OSD will have its main bluestore block on "smithi195-lv-3-data-0qs2g7" 2022-03-10T03:16:11.905 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:11.896082 I | op-osd: skipping OSD prepare job creation for PVC "smithi195-lv-3-data-0qs2g7" because OSD daemon using the PVC already exists 2022-03-10T03:16:11.906 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:11.896085 I | op-osd: OSD will have its main bluestore block on "smithi195-lv-4-data-0v9ftf" 2022-03-10T03:16:11.906 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:11.896088 I | op-osd: skipping OSD prepare job creation for PVC "smithi195-lv-4-data-0v9ftf" because OSD daemon using the PVC already exists 2022-03-10T03:16:11.906 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:11.896091 I | op-osd: OSD will have its main bluestore block on "smithi038-lv-1-data-0rrtwp" 2022-03-10T03:16:11.906 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:11.896095 I | op-osd: skipping OSD prepare job creation for PVC "smithi038-lv-1-data-0rrtwp" because OSD daemon using the PVC already exists 2022-03-10T03:16:11.907 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:11.896099 I | op-osd: start provisioning the OSDs on nodes, if needed 2022-03-10T03:16:11.907 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:11.896104 I | op-osd: no nodes are defined for configuring OSDs on raw devices 2022-03-10T03:16:12.146 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:12.150667 I | op-osd: skipping osd checks for ok-to-stop 2022-03-10T03:16:12.152 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:12.156692 I | op-osd: updating OSD 0 on PVC "smithi038-lv-1-data-0rrtwp" 2022-03-10T03:16:12.153 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:12.156724 I | op-osd: OSD will have its main bluestore block on "smithi038-lv-1-data-0rrtwp" 2022-03-10T03:16:12.314 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:12.318613 I | op-osd: skipping osd checks for ok-to-stop 2022-03-10T03:16:12.321 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:12.325379 I | op-osd: updating OSD 1 on PVC "smithi037-lv-3-data-0z8qlf" 2022-03-10T03:16:12.321 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:12.325409 I | op-osd: OSD will have its main bluestore block on "smithi037-lv-3-data-0z8qlf" 2022-03-10T03:16:12.485 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:12.489137 I | op-osd: skipping osd checks for ok-to-stop 2022-03-10T03:16:12.491 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:12.495889 I | op-osd: updating OSD 10 on PVC "smithi195-lv-2-data-0cgctr" 2022-03-10T03:16:12.491 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:12.495919 I | op-osd: OSD will have its main bluestore block on "smithi195-lv-2-data-0cgctr" 2022-03-10T03:16:12.884 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:12.888480 I | op-mgr: successful modules: dashboard 2022-03-10T03:16:12.890 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:12.894630 I | op-mgr: successful modules: prometheus 2022-03-10T03:16:12.893 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:12.895379 I | op-mgr: successful modules: mgr module(s) from the spec 2022-03-10T03:16:12.970 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:12.974434 I | op-osd: skipping osd checks for ok-to-stop 2022-03-10T03:16:13.063 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:13.067685 I | op-osd: updating OSD 11 on PVC "smithi037-lv-4-data-0pfq7x" 2022-03-10T03:16:13.064 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:13.067719 I | op-osd: OSD will have its main bluestore block on "smithi037-lv-4-data-0pfq7x" 2022-03-10T03:16:13.770 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:13.774538 I | op-osd: skipping osd checks for ok-to-stop 2022-03-10T03:16:13.863 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:13.867651 I | op-osd: updating OSD 2 on PVC "smithi037-lv-2-data-0hgjtd" 2022-03-10T03:16:13.864 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:13.867683 I | op-osd: OSD will have its main bluestore block on "smithi037-lv-2-data-0hgjtd" 2022-03-10T03:16:14.508 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:14.512384 I | op-mgr: successful modules: orchestrator modules 2022-03-10T03:16:14.576 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:14.579805 I | op-osd: skipping osd checks for ok-to-stop 2022-03-10T03:16:14.663 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:14.667664 I | op-osd: updating OSD 3 on PVC "smithi038-lv-4-data-0pzz62" 2022-03-10T03:16:14.664 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:14.667692 I | op-osd: OSD will have its main bluestore block on "smithi038-lv-4-data-0pzz62" 2022-03-10T03:16:15.371 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:15.375826 I | op-osd: skipping osd checks for ok-to-stop 2022-03-10T03:16:15.463 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:15.467316 I | op-osd: updating OSD 4 on PVC "smithi038-lv-3-data-0h4lqd" 2022-03-10T03:16:15.463 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:15.467361 I | op-osd: OSD will have its main bluestore block on "smithi038-lv-3-data-0h4lqd" 2022-03-10T03:16:16.170 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:16.174123 I | op-osd: skipping osd checks for ok-to-stop 2022-03-10T03:16:16.261 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:16.265976 I | op-osd: updating OSD 5 on PVC "smithi038-lv-2-data-0hvtxr" 2022-03-10T03:16:16.262 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:16.266014 I | op-osd: OSD will have its main bluestore block on "smithi038-lv-2-data-0hvtxr" 2022-03-10T03:16:16.971 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:16.975269 I | op-osd: skipping osd checks for ok-to-stop 2022-03-10T03:16:17.063 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:17.067260 I | op-osd: updating OSD 6 on PVC "smithi195-lv-3-data-0qs2g7" 2022-03-10T03:16:17.063 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:17.067304 I | op-osd: OSD will have its main bluestore block on "smithi195-lv-3-data-0qs2g7" 2022-03-10T03:16:17.771 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:17.775636 I | op-osd: skipping osd checks for ok-to-stop 2022-03-10T03:16:17.897 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:17.901290 I | op-osd: updating OSD 7 on PVC "smithi195-lv-1-data-0jjrtk" 2022-03-10T03:16:17.897 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:17.901330 I | op-osd: OSD will have its main bluestore block on "smithi195-lv-1-data-0jjrtk" 2022-03-10T03:16:18.572 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:18.575797 I | op-osd: skipping osd checks for ok-to-stop 2022-03-10T03:16:18.663 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:18.667292 I | op-osd: updating OSD 8 on PVC "smithi037-lv-1-data-08t564" 2022-03-10T03:16:18.664 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:18.667328 I | op-osd: OSD will have its main bluestore block on "smithi037-lv-1-data-08t564" 2022-03-10T03:16:19.371 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:19.374994 I | op-osd: skipping osd checks for ok-to-stop 2022-03-10T03:16:19.463 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:19.467779 I | op-osd: updating OSD 9 on PVC "smithi195-lv-4-data-0v9ftf" 2022-03-10T03:16:19.464 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:19.467821 I | op-osd: OSD will have its main bluestore block on "smithi195-lv-4-data-0v9ftf" 2022-03-10T03:16:20.500 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:20.503961 I | op-osd: finished running OSDs in namespace "rook-ceph" 2022-03-10T03:16:20.500 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:20.503994 I | ceph-cluster-controller: done reconciling ceph cluster in namespace "rook-ceph" 2022-03-10T03:16:24.369 INFO:teuthology.orchestra.run.smithi037.stderr:+ '[' 11 '!=' 12 ']' 2022-03-10T03:16:24.370 INFO:teuthology.orchestra.run.smithi037.stderr:+ echo 'waiting for osd to come back up' 2022-03-10T03:16:24.370 INFO:teuthology.orchestra.run.smithi037.stdout:waiting for osd to come back up 2022-03-10T03:16:24.371 INFO:teuthology.orchestra.run.smithi037.stderr:++ toolbox ceph osd stat 2022-03-10T03:16:24.371 INFO:teuthology.orchestra.run.smithi037.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-03-10T03:16:24.371 INFO:teuthology.orchestra.run.smithi037.stderr:++ cut -f3 -d ' ' 2022-03-10T03:16:24.443 INFO:teuthology.orchestra.run.smithi037.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-10T03:16:24.978 INFO:teuthology.orchestra.run.smithi037.stderr:+ num_osd=12 2022-03-10T03:16:24.978 INFO:teuthology.orchestra.run.smithi037.stderr:+ sleep 30s 2022-03-10T03:16:54.980 INFO:teuthology.orchestra.run.smithi037.stderr:+ '[' 12 '!=' 12 ']' 2022-03-10T03:16:54.982 INFO:teuthology.run_tasks:Running task rook.shell... 2022-03-10T03:16:54.995 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph orch status 2022-03-10T03:16:55.751 INFO:teuthology.orchestra.run.smithi037.stdout:Backend: rook 2022-03-10T03:16:55.752 INFO:teuthology.orchestra.run.smithi037.stdout:Available: Yes 2022-03-10T03:16:55.778 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph orch ps 2022-03-10T03:16:56.338 INFO:teuthology.orchestra.run.smithi037.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID 2022-03-10T03:16:56.339 INFO:teuthology.orchestra.run.smithi037.stdout:crashcollector.crash smithi037 running (3m) 0s ago 3m - - 4b871cde5e1a 2022-03-10T03:16:56.339 INFO:teuthology.orchestra.run.smithi037.stdout:crashcollector.crash smithi038 running (4m) 0s ago 4m - - 4b871cde5e1a 2022-03-10T03:16:56.339 INFO:teuthology.orchestra.run.smithi037.stdout:crashcollector.crash smithi195 running (4m) 0s ago 4m - - 4b871cde5e1a 2022-03-10T03:16:56.339 INFO:teuthology.orchestra.run.smithi037.stdout:mgr.a smithi037 running (5m) 0s ago 5m - - 4b871cde5e1a 2022-03-10T03:16:56.340 INFO:teuthology.orchestra.run.smithi037.stdout:mon.a smithi038 running (7m) 0s ago 7m - - 4b871cde5e1a 2022-03-10T03:16:56.340 INFO:teuthology.orchestra.run.smithi037.stdout:mon.b smithi195 running (6m) 0s ago 6m - - 4b871cde5e1a 2022-03-10T03:16:56.340 INFO:teuthology.orchestra.run.smithi037.stdout:mon.c smithi037 running (5m) 0s ago 5m - - 4b871cde5e1a 2022-03-10T03:16:56.340 INFO:teuthology.orchestra.run.smithi037.stdout:osd.0 smithi038 running (65s) 0s ago 65s - - 4b871cde5e1a 2022-03-10T03:16:56.340 INFO:teuthology.orchestra.run.smithi037.stdout:osd.1 smithi037 running (3m) 0s ago 3m - - 4b871cde5e1a 2022-03-10T03:16:56.341 INFO:teuthology.orchestra.run.smithi037.stdout:osd.10 smithi195 running (3m) 0s ago 3m - - 4b871cde5e1a 2022-03-10T03:16:56.341 INFO:teuthology.orchestra.run.smithi037.stdout:osd.11 smithi037 running (3m) 0s ago 3m - - 4b871cde5e1a 2022-03-10T03:16:56.341 INFO:teuthology.orchestra.run.smithi037.stdout:osd.2 smithi037 running (3m) 0s ago 3m - - 4b871cde5e1a 2022-03-10T03:16:56.342 INFO:teuthology.orchestra.run.smithi037.stdout:osd.3 smithi038 running (3m) 0s ago 3m - - 4b871cde5e1a 2022-03-10T03:16:56.342 INFO:teuthology.orchestra.run.smithi037.stdout:osd.4 smithi038 running (3m) 0s ago 3m - - 4b871cde5e1a 2022-03-10T03:16:56.342 INFO:teuthology.orchestra.run.smithi037.stdout:osd.5 smithi038 running (3m) 0s ago 3m - - 4b871cde5e1a 2022-03-10T03:16:56.342 INFO:teuthology.orchestra.run.smithi037.stdout:osd.6 smithi195 running (3m) 0s ago 3m - - 4b871cde5e1a 2022-03-10T03:16:56.343 INFO:teuthology.orchestra.run.smithi037.stdout:osd.7 smithi195 running (3m) 0s ago 3m - - 4b871cde5e1a 2022-03-10T03:16:56.343 INFO:teuthology.orchestra.run.smithi037.stdout:osd.8 smithi037 running (3m) 0s ago 3m - - 4b871cde5e1a 2022-03-10T03:16:56.343 INFO:teuthology.orchestra.run.smithi037.stdout:osd.9 smithi195 running (3m) 0s ago 3m - - 4b871cde5e1a 2022-03-10T03:16:56.362 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph orch ls 2022-03-10T03:16:57.020 INFO:teuthology.orchestra.run.smithi037.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2022-03-10T03:16:57.020 INFO:teuthology.orchestra.run.smithi037.stdout:crash 3/3 0s ago 4m * 2022-03-10T03:16:57.020 INFO:teuthology.orchestra.run.smithi037.stdout:mgr 1/1 0s ago 5m count:1 2022-03-10T03:16:57.021 INFO:teuthology.orchestra.run.smithi037.stdout:mon 3/3 0s ago 7m count:3 2022-03-10T03:16:57.021 INFO:teuthology.orchestra.run.smithi037.stdout:osd 24 0s ago 3m 2022-03-10T03:16:57.021 INFO:teuthology.orchestra.run.smithi037.stdout:osd.all-available-devices 0 0s ago - * 2022-03-10T03:16:57.043 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph orch host ls 2022-03-10T03:16:57.573 INFO:teuthology.orchestra.run.smithi037.stdout:HOST ADDR LABELS STATUS 2022-03-10T03:16:57.573 INFO:teuthology.orchestra.run.smithi037.stdout:smithi037 172.21.15.37/smithi037 2022-03-10T03:16:57.573 INFO:teuthology.orchestra.run.smithi037.stdout:smithi038 172.21.15.38/smithi038 2022-03-10T03:16:57.574 INFO:teuthology.orchestra.run.smithi037.stdout:smithi195 172.21.15.195/smithi195 2022-03-10T03:16:57.574 INFO:teuthology.orchestra.run.smithi037.stdout:3 hosts in cluster 2022-03-10T03:16:57.593 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph orch device ls 2022-03-10T03:16:58.171 INFO:teuthology.orchestra.run.smithi037.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2022-03-10T03:16:58.171 INFO:teuthology.orchestra.run.smithi037.stdout:smithi037 /dev/vg_nvme/lv_1 unknown None 107G 0s ago 2022-03-10T03:16:58.172 INFO:teuthology.orchestra.run.smithi037.stdout:smithi037 /dev/vg_nvme/lv_2 unknown None 107G 0s ago 2022-03-10T03:16:58.172 INFO:teuthology.orchestra.run.smithi037.stdout:smithi037 /dev/vg_nvme/lv_3 unknown None 107G 0s ago 2022-03-10T03:16:58.172 INFO:teuthology.orchestra.run.smithi037.stdout:smithi037 /dev/vg_nvme/lv_4 unknown None 107G 0s ago 2022-03-10T03:16:58.172 INFO:teuthology.orchestra.run.smithi037.stdout:smithi038 /dev/vg_nvme/lv_1 unknown None 107G 0s ago 2022-03-10T03:16:58.173 INFO:teuthology.orchestra.run.smithi037.stdout:smithi038 /dev/vg_nvme/lv_2 unknown None 107G 0s ago 2022-03-10T03:16:58.173 INFO:teuthology.orchestra.run.smithi037.stdout:smithi038 /dev/vg_nvme/lv_3 unknown None 107G 0s ago 2022-03-10T03:16:58.173 INFO:teuthology.orchestra.run.smithi037.stdout:smithi038 /dev/vg_nvme/lv_4 unknown None 107G 0s ago 2022-03-10T03:16:58.173 INFO:teuthology.orchestra.run.smithi037.stdout:smithi195 /dev/vg_nvme/lv_1 unknown None 107G 0s ago 2022-03-10T03:16:58.173 INFO:teuthology.orchestra.run.smithi037.stdout:smithi195 /dev/vg_nvme/lv_2 unknown None 107G 0s ago 2022-03-10T03:16:58.174 INFO:teuthology.orchestra.run.smithi037.stdout:smithi195 /dev/vg_nvme/lv_3 unknown None 107G 0s ago 2022-03-10T03:16:58.174 INFO:teuthology.orchestra.run.smithi037.stdout:smithi195 /dev/vg_nvme/lv_4 unknown None 107G 0s ago 2022-03-10T03:16:58.192 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph orch apply rgw foo 2022-03-10T03:16:58.774 INFO:teuthology.orchestra.run.smithi037.stdout:Created 2022-03-10T03:16:58.776 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:58.780420 I | ceph-spec: adding finalizer "cephobjectstore.ceph.rook.io" on "foo" 2022-03-10T03:16:58.795 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph orch apply mds foo 2022-03-10T03:16:58.873 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:58.877028 I | op-mon: parsing mon endpoints: a=10.100.186.100:6789,b=10.111.132.151:6789,c=10.101.46.100:6789 2022-03-10T03:16:58.873 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:58.877091 I | ceph-spec: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3... 2022-03-10T03:16:59.396 INFO:teuthology.orchestra.run.smithi037.stdout:Created 2022-03-10T03:16:59.398 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:59.402329 I | ceph-spec: adding finalizer "cephfilesystem.ceph.rook.io" on "foo" 2022-03-10T03:16:59.419 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph orch apply rbd-mirror 2022-03-10T03:16:59.478 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:59.482459 I | op-mon: parsing mon endpoints: a=10.100.186.100:6789,b=10.111.132.151:6789,c=10.101.46.100:6789 2022-03-10T03:16:59.478 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:16:59.482547 I | ceph-spec: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3... 2022-03-10T03:17:00.537 INFO:teuthology.orchestra.run.smithi037.stdout:Success 2022-03-10T03:17:00.559 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph orch apply nfs foo --port 12777 2022-03-10T03:17:00.569 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:17:00.574186 I | op-mon: parsing mon endpoints: a=10.100.186.100:6789,b=10.111.132.151:6789,c=10.101.46.100:6789 2022-03-10T03:17:00.570 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:17:00.574241 I | ceph-spec: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3... 2022-03-10T03:17:03.752 INFO:teuthology.orchestra.run.smithi037.stdout:Unable to create NFS daemon, check logs for more traceback 2022-03-10T03:17:03.753 INFO:teuthology.orchestra.run.smithi037.stdout:'RookOrchestrator' object has no attribute 'open_ioctx' 2022-03-10T03:17:03.778 DEBUG:teuthology.run_tasks:Unwinding manager rook 2022-03-10T03:17:03.792 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph orch ls -f json 2022-03-10T03:17:04.473 INFO:teuthology.orchestra.run.smithi037.stdout: 2022-03-10T03:17:04.473 INFO:teuthology.orchestra.run.smithi037.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:12:03.000000Z", "last_refresh": "2022-03-10T03:17:04.304980Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "mds.foo", "service_type": "mds", "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T03:17:04.304980Z", "running": 0, "size": 2}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:11:53.000000Z", "last_refresh": "2022-03-10T03:17:04.304980Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:09:03.000000Z", "last_refresh": "2022-03-10T03:17:04.304980Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:30.000000Z", "last_refresh": "2022-03-10T03:17:04.304980Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T03:17:04.304980Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_name": "rbd-mirror", "service_type": "rbd-mirror", "status": {"last_refresh": "2022-03-10T03:17:04.304980Z", "running": 0, "size": 1}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T03:17:04.304980Z", "running": 0, "size": 1}}] 2022-03-10T03:17:04.498 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph orch rm mds.foo 2022-03-10T03:17:05.312 INFO:teuthology.orchestra.run.smithi037.stdout:Removed cephfilesystems/foo 2022-03-10T03:17:05.335 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph orch rm rbd-mirror 2022-03-10T03:17:06.088 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:17:06.092476 I | ceph-spec: detected ceph image version: "17.0.0-11025 quincy" 2022-03-10T03:17:06.141 INFO:teuthology.orchestra.run.smithi037.stdout:Removed cephrbdmirrors/default-rbd-mirror 2022-03-10T03:17:06.162 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph orch rm rgw.foo 2022-03-10T03:17:06.389 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:17:06.393468 I | ceph-spec: detected ceph image version: "17.0.0-11025 quincy" 2022-03-10T03:17:06.530 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:17:06.533963 W | ceph-rbd-mirror-controller: (DEPRECATED) use of peer secret names in CephRBDMirror is deprecated. Please use CephBlockPool CR to configure peer secret names and import peers. 2022-03-10T03:17:06.531 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:17:06.533996 I | ceph-rbd-mirror-controller: configure rbd-mirroring with 1 workers 2022-03-10T03:17:06.531 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:17:06.534009 I | cephclient: getting or creating ceph auth key "client.rbd-mirror.a" 2022-03-10T03:17:06.890 INFO:teuthology.orchestra.run.smithi037.stdout:Removed cephobjectstores/foo 2022-03-10T03:17:06.919 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph orch ls -f json 2022-03-10T03:17:07.195 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:17:07.198810 I | ceph-rbd-mirror-controller: "rook-ceph-rbd-mirror-a" deployment started 2022-03-10T03:17:07.462 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:17:07.466053 I | ceph-spec: object "rook-ceph-rbd-mirror-a" matched on delete, reconciling 2022-03-10T03:17:07.874 INFO:teuthology.orchestra.run.smithi037.stdout: 2022-03-10T03:17:07.875 INFO:teuthology.orchestra.run.smithi037.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:12:03.000000Z", "last_refresh": "2022-03-10T03:17:07.501454Z", "running": 4, "size": 3}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "mds.foo", "service_type": "mds", "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T03:17:07.501454Z", "running": 0, "size": 2}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:11:53.000000Z", "last_refresh": "2022-03-10T03:17:07.501454Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:09:03.000000Z", "last_refresh": "2022-03-10T03:17:07.501454Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:30.000000Z", "last_refresh": "2022-03-10T03:17:07.501454Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T03:17:07.501454Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T03:17:07.501454Z", "running": 0, "size": 1}}] 2022-03-10T03:17:07.969 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:17:07.972533 I | ceph-spec: detected ceph image version: "17.0.0-11025 quincy" 2022-03-10T03:17:08.005 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:17:08.009089 I | ceph-file-controller: start running mdses for filesystem "foo" 2022-03-10T03:17:08.432 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:17:08.436169 I | cephclient: getting or creating ceph auth key "mds.foo-a" 2022-03-10T03:17:09.417 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:17:09.421198 I | op-mds: setting mds config flags 2022-03-10T03:17:09.417 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:17:09.421220 I | op-config: setting "mds.foo-a"="mds_join_fs"="foo" option to the mon configuration database 2022-03-10T03:17:09.676 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:17:09.679992 I | ceph-object-controller: reconciling object store deployments 2022-03-10T03:17:09.715 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:17:09.718040 I | ceph-object-controller: ceph object store gateway service running at 10.111.185.253 2022-03-10T03:17:09.716 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:17:09.718076 I | ceph-object-controller: reconciling object store pools 2022-03-10T03:17:10.050 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:17:10.054670 I | op-config: successfully set "mds.foo-a"="mds_join_fs"="foo" option to the mon configuration database 2022-03-10T03:17:10.080 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:17:10.083984 W | ceph-object-controller: failed to adjust the PG count for rgw metadata pools. using the general default. failed to get config setting "rgw_rados_pool_pg_num_min" for user "mon.": exit status 2 2022-03-10T03:17:10.126 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:17:10.126833 I | cephclient: getting or creating ceph auth key "mds.foo-b" 2022-03-10T03:17:10.448 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:17:10.452849 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi195": the object has been modified; please apply your changes to the latest version and try again 2022-03-10T03:17:11.809 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:17:11.813258 I | op-mds: setting mds config flags 2022-03-10T03:17:11.809 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:17:11.813287 I | op-config: setting "mds.foo-b"="mds_join_fs"="foo" option to the mon configuration database 2022-03-10T03:17:12.430 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:17:12.433920 I | op-config: successfully set "mds.foo-b"="mds_join_fs"="foo" option to the mon configuration database 2022-03-10T03:17:14.485 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:17:14.488612 I | ceph-file-controller: creating filesystem "foo" 2022-03-10T03:17:17.893 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph orch ls -f json 2022-03-10T03:17:19.223 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:17:19.227163 I | cephclient: reconciling replicated pool foo.rgw.buckets.index succeeded 2022-03-10T03:17:21.127 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:17:21.129505 I | cephclient: reconciling replicated pool foo.rgw.control succeeded 2022-03-10T03:17:21.128 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:17:21.130349 I | cephclient: reconciling replicated pool foo.rgw.buckets.non-ec succeeded 2022-03-10T03:17:21.128 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:17:21.130533 I | cephclient: reconciling replicated pool foo-metadata succeeded 2022-03-10T03:17:21.129 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:17:21.130588 I | cephclient: reconciling replicated pool foo.rgw.log succeeded 2022-03-10T03:17:21.129 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:17:21.130714 I | cephclient: reconciling replicated pool foo.rgw.meta succeeded 2022-03-10T03:17:21.953 INFO:teuthology.orchestra.run.smithi037.stdout: 2022-03-10T03:17:21.954 INFO:teuthology.orchestra.run.smithi037.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:12:19.000000Z", "last_refresh": "2022-03-10T03:17:21.757645Z", "running": 4, "size": 3}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "mds.foo", "service_type": "mds", "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T03:17:21.757645Z", "running": 0, "size": 2}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:11:53.000000Z", "last_refresh": "2022-03-10T03:17:21.757645Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:09:03.000000Z", "last_refresh": "2022-03-10T03:17:21.757645Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:30.000000Z", "last_refresh": "2022-03-10T03:17:21.757645Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T03:17:21.757645Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T03:17:21.757645Z", "running": 0, "size": 1}}] 2022-03-10T03:17:22.337 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:17:22.341275 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool ".rgw.root": failed to tag pool ".rgw.root" for application "rook-ceph-rgw": failed to get application for pool ".rgw.root": failed to get current application for pool .rgw.root: exit status 2 2022-03-10T03:17:22.346 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:17:22.350982 I | op-mon: parsing mon endpoints: a=10.100.186.100:6789,b=10.111.132.151:6789,c=10.101.46.100:6789 2022-03-10T03:17:24.368 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:17:24.372207 I | cephclient: reconciling replicated pool foo-data0 succeeded 2022-03-10T03:17:25.096 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:17:25.099833 I | cephclient: creating filesystem "foo" with metadata pool "foo-metadata" and data pools [foo-data0] 2022-03-10T03:17:25.697 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:17:25.700851 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T03:17:25.693+0000 7f415bbd7380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T03:17:25.698 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10T03:17:25.693+0000 7f415bbd7380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T03:17:25.698 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10T03:17:25.693+0000 7f415bbd7380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T03:17:25.698 INFO:tasks.rook.operator.smithi037.stdout:couldn't init storage provider. : exit status 5 2022-03-10T03:17:25.713 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:17:25.717746 I | op-mon: parsing mon endpoints: a=10.100.186.100:6789,b=10.111.132.151:6789,c=10.101.46.100:6789 2022-03-10T03:17:26.291 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:17:26.294594 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T03:17:26.285+0000 7f2a0819b380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T03:17:26.291 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10T03:17:26.285+0000 7f2a0819b380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T03:17:26.291 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10T03:17:26.285+0000 7f2a0819b380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T03:17:26.292 INFO:tasks.rook.operator.smithi037.stdout:couldn't init storage provider. : exit status 5 2022-03-10T03:17:26.305 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:17:26.309464 I | op-mon: parsing mon endpoints: a=10.100.186.100:6789,b=10.111.132.151:6789,c=10.101.46.100:6789 2022-03-10T03:17:27.053 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:17:27.056955 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T03:17:27.049+0000 7f036b054380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T03:17:27.054 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10T03:17:27.049+0000 7f036b054380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T03:17:27.054 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10T03:17:27.049+0000 7f036b054380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T03:17:27.054 INFO:tasks.rook.operator.smithi037.stdout:couldn't init storage provider. : exit status 5 2022-03-10T03:17:27.063 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:17:27.067508 I | op-mon: parsing mon endpoints: a=10.100.186.100:6789,b=10.111.132.151:6789,c=10.101.46.100:6789 2022-03-10T03:17:27.478 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:17:27.478165 I | ceph-file-controller: created filesystem "foo" on 1 data pool(s) and metadata pool "foo-metadata" 2022-03-10T03:17:27.478 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:17:27.478198 I | cephclient: setting allow_standby_replay for filesystem "foo" 2022-03-10T03:17:27.737 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:17:27.740882 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T03:17:27.733+0000 7f863741d380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T03:17:27.738 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10T03:17:27.733+0000 7f863741d380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T03:17:27.738 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10T03:17:27.733+0000 7f863741d380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T03:17:27.738 INFO:tasks.rook.operator.smithi037.stdout:couldn't init storage provider. : exit status 5 2022-03-10T03:17:27.764 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:17:27.768292 I | op-mon: parsing mon endpoints: a=10.100.186.100:6789,b=10.111.132.151:6789,c=10.101.46.100:6789 2022-03-10T03:17:28.378 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:17:28.382280 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T03:17:28.373+0000 7f2ca8aff380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T03:17:28.379 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10T03:17:28.373+0000 7f2ca8aff380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T03:17:28.379 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10T03:17:28.373+0000 7f2ca8aff380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T03:17:28.379 INFO:tasks.rook.operator.smithi037.stdout:couldn't init storage provider. : exit status 5 2022-03-10T03:17:28.397 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:17:28.401062 I | op-mon: parsing mon endpoints: a=10.100.186.100:6789,b=10.111.132.151:6789,c=10.101.46.100:6789 2022-03-10T03:17:28.711 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:17:28.715112 I | op-mon: parsing mon endpoints: a=10.100.186.100:6789,b=10.111.132.151:6789,c=10.101.46.100:6789 2022-03-10T03:17:28.722 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:17:28.725429 I | ceph-file-controller: CephFilesystem "rook-ceph/foo" can be deleted safely. deleting CephFilesystem "rook-ceph/foo" 2022-03-10T03:17:29.072 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:17:29.075458 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T03:17:29.065+0000 7f2b749aa380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T03:17:29.072 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10T03:17:29.065+0000 7f2b749aa380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T03:17:29.072 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10T03:17:29.065+0000 7f2b749aa380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T03:17:29.072 INFO:tasks.rook.operator.smithi037.stdout:couldn't init storage provider. : exit status 5 2022-03-10T03:17:29.088 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:17:29.092750 I | op-mon: parsing mon endpoints: a=10.100.186.100:6789,b=10.111.132.151:6789,c=10.101.46.100:6789 2022-03-10T03:17:29.571 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:17:29.575084 I | op-config: deleting "mds_join_fs" option from the mon configuration database 2022-03-10T03:17:29.672 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:17:29.676053 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T03:17:29.669+0000 7f02aa155380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T03:17:29.673 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10T03:17:29.669+0000 7f02aa155380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T03:17:29.673 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10T03:17:29.669+0000 7f02aa155380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T03:17:29.673 INFO:tasks.rook.operator.smithi037.stdout:couldn't init storage provider. : exit status 5 2022-03-10T03:17:29.688 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:17:29.692281 I | op-mon: parsing mon endpoints: a=10.100.186.100:6789,b=10.111.132.151:6789,c=10.101.46.100:6789 2022-03-10T03:17:30.199 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:17:30.203337 I | op-config: successfully deleted "mds_join_fs" option from the mon configuration database 2022-03-10T03:17:30.200 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:17:30.203372 I | op-mds: successfully deleted mds config for "mds.foo-a" in mon configuration database 2022-03-10T03:17:30.200 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:17:30.203382 I | cephclient: deleting ceph auth "mds.foo-a" 2022-03-10T03:17:30.352 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:17:30.355353 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T03:17:30.345+0000 7ff4c42c7380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T03:17:30.352 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10T03:17:30.345+0000 7ff4c42c7380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T03:17:30.352 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10T03:17:30.345+0000 7ff4c42c7380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T03:17:30.353 INFO:tasks.rook.operator.smithi037.stdout:couldn't init storage provider. : exit status 5 2022-03-10T03:17:30.376 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:17:30.380313 I | op-mon: parsing mon endpoints: a=10.100.186.100:6789,b=10.111.132.151:6789,c=10.101.46.100:6789 2022-03-10T03:17:30.739 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:17:30.743059 I | op-mds: successfully deleted mds CephX key for "mds.foo-a" 2022-03-10T03:17:30.982 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:17:30.985925 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T03:17:30.973+0000 7f73758b1380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T03:17:30.983 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10T03:17:30.973+0000 7f73758b1380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T03:17:30.983 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10T03:17:30.973+0000 7f73758b1380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T03:17:30.983 INFO:tasks.rook.operator.smithi037.stdout:couldn't init storage provider. : exit status 5 2022-03-10T03:17:31.173 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:17:31.177547 I | op-config: deleting "mds_join_fs" option from the mon configuration database 2022-03-10T03:17:31.639 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:17:31.643244 I | op-config: successfully deleted "mds_join_fs" option from the mon configuration database 2022-03-10T03:17:31.640 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:17:31.643275 I | op-mds: successfully deleted mds config for "mds.foo-b" in mon configuration database 2022-03-10T03:17:31.640 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:17:31.643284 I | cephclient: deleting ceph auth "mds.foo-b" 2022-03-10T03:17:31.655 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:17:31.654504 I | op-mon: parsing mon endpoints: a=10.100.186.100:6789,b=10.111.132.151:6789,c=10.101.46.100:6789 2022-03-10T03:17:31.978 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph orch ls -f json 2022-03-10T03:17:32.353 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:17:32.356646 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T03:17:32.349+0000 7f1b1a86f380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T03:17:32.354 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10T03:17:32.349+0000 7f1b1a86f380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T03:17:32.354 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10T03:17:32.349+0000 7f1b1a86f380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T03:17:32.354 INFO:tasks.rook.operator.smithi037.stdout:couldn't init storage provider. : exit status 5 2022-03-10T03:17:32.364 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:17:32.368688 I | op-mds: successfully deleted mds CephX key for "mds.foo-b" 2022-03-10T03:17:32.365 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:17:32.368718 I | ceph-file-controller: downing filesystem "foo" 2022-03-10T03:17:32.698 INFO:teuthology.orchestra.run.smithi037.stdout: 2022-03-10T03:17:32.698 INFO:teuthology.orchestra.run.smithi037.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:31.000000Z", "last_refresh": "2022-03-10T03:17:32.538770Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "mds.foo", "service_type": "mds", "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T03:17:32.538770Z", "running": 0, "size": 2}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:11:53.000000Z", "last_refresh": "2022-03-10T03:17:32.538770Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:09:03.000000Z", "last_refresh": "2022-03-10T03:17:32.538770Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:30.000000Z", "last_refresh": "2022-03-10T03:17:32.538770Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T03:17:32.538770Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T03:17:32.538770Z", "running": 0, "size": 1}}] 2022-03-10T03:17:33.446 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:17:33.449788 I | ceph-file-controller: downed filesystem "foo" 2022-03-10T03:17:35.742 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:17:35.746480 I | cephclient: no images/snapshosts present in pool "foo-metadata" 2022-03-10T03:17:35.743 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:17:35.746507 I | cephclient: purging pool "foo-metadata" (id=8) 2022-03-10T03:17:37.486 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:17:37.489315 I | op-mon: parsing mon endpoints: a=10.100.186.100:6789,b=10.111.132.151:6789,c=10.101.46.100:6789 2022-03-10T03:17:38.209 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:17:38.211871 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T03:17:38.204+0000 7fbc7df07380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T03:17:38.210 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10T03:17:38.204+0000 7fbc7df07380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T03:17:38.210 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10T03:17:38.204+0000 7fbc7df07380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T03:17:38.211 INFO:tasks.rook.operator.smithi037.stdout:couldn't init storage provider. : exit status 5 2022-03-10T03:17:39.303 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:17:39.306979 I | cephclient: purge completed for pool "foo-metadata" 2022-03-10T03:17:39.713 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:17:39.716637 I | cephclient: no images/snapshosts present in pool "foo-data0" 2022-03-10T03:17:39.713 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:17:39.716663 I | cephclient: purging pool "foo-data0" (id=10) 2022-03-10T03:17:41.520 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:17:41.523907 I | cephclient: purge completed for pool "foo-data0" 2022-03-10T03:17:41.527 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:17:41.523996 I | ceph-spec: removing finalizer "cephfilesystem.ceph.rook.io" on "foo" 2022-03-10T03:17:41.563 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:17:41.567281 I | ceph-spec: object "rook-ceph-mds-foo-b-keyring" matched on delete, reconciling 2022-03-10T03:17:41.589 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:17:41.592862 I | ceph-spec: object "rook-ceph-mds-foo-b" matched on delete, reconciling 2022-03-10T03:17:41.590 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:17:41.593013 I | ceph-spec: object "rook-ceph-mds-foo-a" matched on delete, reconciling 2022-03-10T03:17:41.590 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:17:41.593233 I | ceph-spec: object "rook-ceph-mds-foo-a-keyring" matched on delete, reconciling 2022-03-10T03:17:42.724 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph orch ls -f json 2022-03-10T03:17:43.471 INFO:teuthology.orchestra.run.smithi037.stdout: 2022-03-10T03:17:43.472 INFO:teuthology.orchestra.run.smithi037.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:31.000000Z", "last_refresh": "2022-03-10T03:17:43.271542Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:11:53.000000Z", "last_refresh": "2022-03-10T03:17:43.271542Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:09:03.000000Z", "last_refresh": "2022-03-10T03:17:43.271542Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:30.000000Z", "last_refresh": "2022-03-10T03:17:43.271542Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T03:17:43.271542Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T03:17:43.271542Z", "running": 0, "size": 1}}] 2022-03-10T03:17:45.520 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:17:45.524506 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi195": the object has been modified; please apply your changes to the latest version and try again 2022-03-10T03:17:48.457 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:17:48.460584 I | op-mon: parsing mon endpoints: a=10.100.186.100:6789,b=10.111.132.151:6789,c=10.101.46.100:6789 2022-03-10T03:17:49.091 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:17:49.094575 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T03:17:49.084+0000 7f7adbe5e380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T03:17:49.091 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10T03:17:49.084+0000 7f7adbe5e380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T03:17:49.092 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10T03:17:49.084+0000 7f7adbe5e380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T03:17:49.092 INFO:tasks.rook.operator.smithi037.stdout:couldn't init storage provider. : exit status 5 2022-03-10T03:17:53.501 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph orch ls -f json 2022-03-10T03:17:54.176 INFO:teuthology.orchestra.run.smithi037.stdout: 2022-03-10T03:17:54.177 INFO:teuthology.orchestra.run.smithi037.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:31.000000Z", "last_refresh": "2022-03-10T03:17:54.001003Z", "running": 4, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:11:53.000000Z", "last_refresh": "2022-03-10T03:17:54.001003Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:09:03.000000Z", "last_refresh": "2022-03-10T03:17:54.001003Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:30.000000Z", "last_refresh": "2022-03-10T03:17:54.001003Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T03:17:54.001003Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T03:17:54.001003Z", "running": 0, "size": 1}}] 2022-03-10T03:18:04.204 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph orch ls -f json 2022-03-10T03:18:04.939 INFO:teuthology.orchestra.run.smithi037.stdout: 2022-03-10T03:18:04.939 INFO:teuthology.orchestra.run.smithi037.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:31.000000Z", "last_refresh": "2022-03-10T03:18:04.767555Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:11:53.000000Z", "last_refresh": "2022-03-10T03:18:04.767555Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:09:03.000000Z", "last_refresh": "2022-03-10T03:18:04.767555Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:30.000000Z", "last_refresh": "2022-03-10T03:18:04.767555Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T03:18:04.767555Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T03:18:04.767555Z", "running": 0, "size": 1}}] 2022-03-10T03:18:09.580 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:18:09.584138 I | op-mon: parsing mon endpoints: a=10.100.186.100:6789,b=10.111.132.151:6789,c=10.101.46.100:6789 2022-03-10T03:18:10.165 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:18:10.168402 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T03:18:10.160+0000 7f3fb420e380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T03:18:10.165 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10T03:18:10.160+0000 7f3fb420e380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T03:18:10.166 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10T03:18:10.160+0000 7f3fb420e380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T03:18:10.166 INFO:tasks.rook.operator.smithi037.stdout:couldn't init storage provider. : exit status 5 2022-03-10T03:18:14.966 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph orch ls -f json 2022-03-10T03:18:15.720 INFO:teuthology.orchestra.run.smithi037.stdout: 2022-03-10T03:18:15.720 INFO:teuthology.orchestra.run.smithi037.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:31.000000Z", "last_refresh": "2022-03-10T03:18:15.533141Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:11:53.000000Z", "last_refresh": "2022-03-10T03:18:15.533141Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:09:03.000000Z", "last_refresh": "2022-03-10T03:18:15.533141Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:30.000000Z", "last_refresh": "2022-03-10T03:18:15.533141Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T03:18:15.533141Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T03:18:15.533141Z", "running": 0, "size": 1}}] 2022-03-10T03:18:25.766 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph orch ls -f json 2022-03-10T03:18:26.539 INFO:teuthology.orchestra.run.smithi037.stdout: 2022-03-10T03:18:26.539 INFO:teuthology.orchestra.run.smithi037.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:31.000000Z", "last_refresh": "2022-03-10T03:18:26.340497Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:11:53.000000Z", "last_refresh": "2022-03-10T03:18:26.340497Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:09:03.000000Z", "last_refresh": "2022-03-10T03:18:26.340497Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:30.000000Z", "last_refresh": "2022-03-10T03:18:26.340497Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T03:18:26.340497Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T03:18:26.340497Z", "running": 0, "size": 1}}] 2022-03-10T03:18:36.566 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph orch ls -f json 2022-03-10T03:18:37.310 INFO:teuthology.orchestra.run.smithi037.stdout: 2022-03-10T03:18:37.310 INFO:teuthology.orchestra.run.smithi037.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:31.000000Z", "last_refresh": "2022-03-10T03:18:37.135679Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:11:53.000000Z", "last_refresh": "2022-03-10T03:18:37.135679Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:09:03.000000Z", "last_refresh": "2022-03-10T03:18:37.135679Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:30.000000Z", "last_refresh": "2022-03-10T03:18:37.135679Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T03:18:37.135679Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T03:18:37.135679Z", "running": 0, "size": 1}}] 2022-03-10T03:18:47.339 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph orch ls -f json 2022-03-10T03:18:48.088 INFO:teuthology.orchestra.run.smithi037.stdout: 2022-03-10T03:18:48.089 INFO:teuthology.orchestra.run.smithi037.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:31.000000Z", "last_refresh": "2022-03-10T03:18:47.910299Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:11:53.000000Z", "last_refresh": "2022-03-10T03:18:47.910299Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:09:03.000000Z", "last_refresh": "2022-03-10T03:18:47.910299Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:30.000000Z", "last_refresh": "2022-03-10T03:18:47.910299Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T03:18:47.910299Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T03:18:47.910299Z", "running": 0, "size": 1}}] 2022-03-10T03:18:51.136 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:18:51.139867 I | op-mon: parsing mon endpoints: a=10.100.186.100:6789,b=10.111.132.151:6789,c=10.101.46.100:6789 2022-03-10T03:18:51.845 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:18:51.848269 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T03:18:51.835+0000 7f7496518380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T03:18:51.846 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10T03:18:51.835+0000 7f7496518380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T03:18:51.846 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10T03:18:51.835+0000 7f7496518380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T03:18:51.846 INFO:tasks.rook.operator.smithi037.stdout:couldn't init storage provider. : exit status 5 2022-03-10T03:18:58.115 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph orch ls -f json 2022-03-10T03:18:58.906 INFO:teuthology.orchestra.run.smithi037.stdout: 2022-03-10T03:18:58.906 INFO:teuthology.orchestra.run.smithi037.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:31.000000Z", "last_refresh": "2022-03-10T03:18:58.733684Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:11:53.000000Z", "last_refresh": "2022-03-10T03:18:58.733684Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:09:03.000000Z", "last_refresh": "2022-03-10T03:18:58.733684Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:30.000000Z", "last_refresh": "2022-03-10T03:18:58.733684Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T03:18:58.733684Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T03:18:58.733684Z", "running": 0, "size": 1}}] 2022-03-10T03:19:08.934 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph orch ls -f json 2022-03-10T03:19:09.679 INFO:teuthology.orchestra.run.smithi037.stdout: 2022-03-10T03:19:09.680 INFO:teuthology.orchestra.run.smithi037.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:31.000000Z", "last_refresh": "2022-03-10T03:19:09.496183Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:11:53.000000Z", "last_refresh": "2022-03-10T03:19:09.496183Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:09:03.000000Z", "last_refresh": "2022-03-10T03:19:09.496183Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:30.000000Z", "last_refresh": "2022-03-10T03:19:09.496183Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T03:19:09.496183Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T03:19:09.496183Z", "running": 0, "size": 1}}] 2022-03-10T03:19:12.520 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:19:12.523465 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi038": the object has been modified; please apply your changes to the latest version and try again 2022-03-10T03:19:19.707 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph orch ls -f json 2022-03-10T03:19:20.424 INFO:teuthology.orchestra.run.smithi037.stdout: 2022-03-10T03:19:20.425 INFO:teuthology.orchestra.run.smithi037.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:31.000000Z", "last_refresh": "2022-03-10T03:19:20.259298Z", "running": 4, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:11:53.000000Z", "last_refresh": "2022-03-10T03:19:20.259298Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:09:03.000000Z", "last_refresh": "2022-03-10T03:19:20.259298Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:30.000000Z", "last_refresh": "2022-03-10T03:19:20.259298Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T03:19:20.259298Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T03:19:20.259298Z", "running": 0, "size": 1}}] 2022-03-10T03:19:30.451 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph orch ls -f json 2022-03-10T03:19:31.145 INFO:teuthology.orchestra.run.smithi037.stdout: 2022-03-10T03:19:31.146 INFO:teuthology.orchestra.run.smithi037.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:31.000000Z", "last_refresh": "2022-03-10T03:19:30.973849Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:11:53.000000Z", "last_refresh": "2022-03-10T03:19:30.973849Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:09:03.000000Z", "last_refresh": "2022-03-10T03:19:30.973849Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:30.000000Z", "last_refresh": "2022-03-10T03:19:30.973849Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T03:19:30.973849Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T03:19:30.973849Z", "running": 0, "size": 1}}] 2022-03-10T03:19:41.173 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph orch ls -f json 2022-03-10T03:19:41.938 INFO:teuthology.orchestra.run.smithi037.stdout: 2022-03-10T03:19:41.938 INFO:teuthology.orchestra.run.smithi037.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:31.000000Z", "last_refresh": "2022-03-10T03:19:41.746323Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:11:53.000000Z", "last_refresh": "2022-03-10T03:19:41.746323Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:09:03.000000Z", "last_refresh": "2022-03-10T03:19:41.746323Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:30.000000Z", "last_refresh": "2022-03-10T03:19:41.746323Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T03:19:41.746323Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T03:19:41.746323Z", "running": 0, "size": 1}}] 2022-03-10T03:19:51.967 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph orch ls -f json 2022-03-10T03:19:52.985 INFO:teuthology.orchestra.run.smithi037.stdout: 2022-03-10T03:19:52.985 INFO:teuthology.orchestra.run.smithi037.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:31.000000Z", "last_refresh": "2022-03-10T03:19:52.546079Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:11:53.000000Z", "last_refresh": "2022-03-10T03:19:52.546079Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:09:03.000000Z", "last_refresh": "2022-03-10T03:19:52.546079Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:30.000000Z", "last_refresh": "2022-03-10T03:19:52.546079Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T03:19:52.546079Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T03:19:52.546079Z", "running": 0, "size": 1}}] 2022-03-10T03:20:03.013 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph orch ls -f json 2022-03-10T03:20:03.757 INFO:teuthology.orchestra.run.smithi037.stdout: 2022-03-10T03:20:03.758 INFO:teuthology.orchestra.run.smithi037.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:31.000000Z", "last_refresh": "2022-03-10T03:20:03.574811Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:11:53.000000Z", "last_refresh": "2022-03-10T03:20:03.574811Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:09:03.000000Z", "last_refresh": "2022-03-10T03:20:03.574811Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:30.000000Z", "last_refresh": "2022-03-10T03:20:03.574811Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T03:20:03.574811Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T03:20:03.574811Z", "running": 0, "size": 1}}] 2022-03-10T03:20:13.774 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:20:13.777028 I | op-mon: parsing mon endpoints: a=10.100.186.100:6789,b=10.111.132.151:6789,c=10.101.46.100:6789 2022-03-10T03:20:13.782 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph orch ls -f json 2022-03-10T03:20:14.468 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:20:14.471325 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T03:20:14.461+0000 7f816e4f7380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T03:20:14.469 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10T03:20:14.461+0000 7f816e4f7380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T03:20:14.469 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10T03:20:14.461+0000 7f816e4f7380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T03:20:14.469 INFO:tasks.rook.operator.smithi037.stdout:couldn't init storage provider. : exit status 5 2022-03-10T03:20:14.499 INFO:teuthology.orchestra.run.smithi037.stdout: 2022-03-10T03:20:14.499 INFO:teuthology.orchestra.run.smithi037.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:31.000000Z", "last_refresh": "2022-03-10T03:20:14.328517Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:11:53.000000Z", "last_refresh": "2022-03-10T03:20:14.328517Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:09:03.000000Z", "last_refresh": "2022-03-10T03:20:14.328517Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:30.000000Z", "last_refresh": "2022-03-10T03:20:14.328517Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T03:20:14.328517Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T03:20:14.328517Z", "running": 0, "size": 1}}] 2022-03-10T03:20:24.526 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph orch ls -f json 2022-03-10T03:20:25.207 INFO:teuthology.orchestra.run.smithi037.stdout: 2022-03-10T03:20:25.207 INFO:teuthology.orchestra.run.smithi037.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:31.000000Z", "last_refresh": "2022-03-10T03:20:25.042856Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:11:53.000000Z", "last_refresh": "2022-03-10T03:20:25.042856Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:09:03.000000Z", "last_refresh": "2022-03-10T03:20:25.042856Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:30.000000Z", "last_refresh": "2022-03-10T03:20:25.042856Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T03:20:25.042856Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T03:20:25.042856Z", "running": 0, "size": 1}}] 2022-03-10T03:20:35.235 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph orch ls -f json 2022-03-10T03:20:35.964 INFO:teuthology.orchestra.run.smithi037.stdout: 2022-03-10T03:20:35.965 INFO:teuthology.orchestra.run.smithi037.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:31.000000Z", "last_refresh": "2022-03-10T03:20:35.794788Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:11:53.000000Z", "last_refresh": "2022-03-10T03:20:35.794788Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:09:03.000000Z", "last_refresh": "2022-03-10T03:20:35.794788Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:30.000000Z", "last_refresh": "2022-03-10T03:20:35.794788Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T03:20:35.794788Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T03:20:35.794788Z", "running": 0, "size": 1}}] 2022-03-10T03:20:45.991 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph orch ls -f json 2022-03-10T03:20:46.856 INFO:teuthology.orchestra.run.smithi037.stdout: 2022-03-10T03:20:46.856 INFO:teuthology.orchestra.run.smithi037.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:31.000000Z", "last_refresh": "2022-03-10T03:20:46.623650Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:11:53.000000Z", "last_refresh": "2022-03-10T03:20:46.623650Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:09:03.000000Z", "last_refresh": "2022-03-10T03:20:46.623650Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:30.000000Z", "last_refresh": "2022-03-10T03:20:46.623650Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T03:20:46.623650Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T03:20:46.623650Z", "running": 0, "size": 1}}] 2022-03-10T03:20:56.884 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph orch ls -f json 2022-03-10T03:20:57.601 INFO:teuthology.orchestra.run.smithi037.stdout: 2022-03-10T03:20:57.601 INFO:teuthology.orchestra.run.smithi037.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:31.000000Z", "last_refresh": "2022-03-10T03:20:57.432075Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:11:53.000000Z", "last_refresh": "2022-03-10T03:20:57.432075Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:09:03.000000Z", "last_refresh": "2022-03-10T03:20:57.432075Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:30.000000Z", "last_refresh": "2022-03-10T03:20:57.432075Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T03:20:57.432075Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T03:20:57.432075Z", "running": 0, "size": 1}}] 2022-03-10T03:21:07.628 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph orch ls -f json 2022-03-10T03:21:08.364 INFO:teuthology.orchestra.run.smithi037.stdout: 2022-03-10T03:21:08.365 INFO:teuthology.orchestra.run.smithi037.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:31.000000Z", "last_refresh": "2022-03-10T03:21:08.178198Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:11:53.000000Z", "last_refresh": "2022-03-10T03:21:08.178198Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:09:03.000000Z", "last_refresh": "2022-03-10T03:21:08.178198Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:30.000000Z", "last_refresh": "2022-03-10T03:21:08.178198Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T03:21:08.178198Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T03:21:08.178198Z", "running": 0, "size": 1}}] 2022-03-10T03:21:18.389 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph orch ls -f json 2022-03-10T03:21:19.211 INFO:teuthology.orchestra.run.smithi037.stdout: 2022-03-10T03:21:19.211 INFO:teuthology.orchestra.run.smithi037.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:31.000000Z", "last_refresh": "2022-03-10T03:21:18.989905Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:11:53.000000Z", "last_refresh": "2022-03-10T03:21:18.989905Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:09:03.000000Z", "last_refresh": "2022-03-10T03:21:18.989905Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:30.000000Z", "last_refresh": "2022-03-10T03:21:18.989905Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T03:21:18.989905Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T03:21:18.989905Z", "running": 0, "size": 1}}] 2022-03-10T03:21:29.238 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph orch ls -f json 2022-03-10T03:21:29.952 INFO:teuthology.orchestra.run.smithi037.stdout: 2022-03-10T03:21:29.952 INFO:teuthology.orchestra.run.smithi037.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:31.000000Z", "last_refresh": "2022-03-10T03:21:29.777120Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:11:53.000000Z", "last_refresh": "2022-03-10T03:21:29.777120Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:09:03.000000Z", "last_refresh": "2022-03-10T03:21:29.777120Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:30.000000Z", "last_refresh": "2022-03-10T03:21:29.777120Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T03:21:29.777120Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T03:21:29.777120Z", "running": 0, "size": 1}}] 2022-03-10T03:21:39.977 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph orch ls -f json 2022-03-10T03:21:40.702 INFO:teuthology.orchestra.run.smithi037.stdout: 2022-03-10T03:21:40.703 INFO:teuthology.orchestra.run.smithi037.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:31.000000Z", "last_refresh": "2022-03-10T03:21:40.522709Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:11:53.000000Z", "last_refresh": "2022-03-10T03:21:40.522709Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:09:03.000000Z", "last_refresh": "2022-03-10T03:21:40.522709Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:30.000000Z", "last_refresh": "2022-03-10T03:21:40.522709Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T03:21:40.522709Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T03:21:40.522709Z", "running": 0, "size": 1}}] 2022-03-10T03:21:50.729 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph orch ls -f json 2022-03-10T03:21:51.458 INFO:teuthology.orchestra.run.smithi037.stdout: 2022-03-10T03:21:51.459 INFO:teuthology.orchestra.run.smithi037.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:31.000000Z", "last_refresh": "2022-03-10T03:21:51.291224Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:11:53.000000Z", "last_refresh": "2022-03-10T03:21:51.291224Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:09:03.000000Z", "last_refresh": "2022-03-10T03:21:51.291224Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:30.000000Z", "last_refresh": "2022-03-10T03:21:51.291224Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T03:21:51.291224Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T03:21:51.291224Z", "running": 0, "size": 1}}] 2022-03-10T03:22:01.486 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph orch ls -f json 2022-03-10T03:22:02.563 INFO:teuthology.orchestra.run.smithi037.stdout: 2022-03-10T03:22:02.564 INFO:teuthology.orchestra.run.smithi037.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:31.000000Z", "last_refresh": "2022-03-10T03:22:02.361473Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:11:53.000000Z", "last_refresh": "2022-03-10T03:22:02.361473Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:09:03.000000Z", "last_refresh": "2022-03-10T03:22:02.361473Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:30.000000Z", "last_refresh": "2022-03-10T03:22:02.361473Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T03:22:02.361473Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T03:22:02.361473Z", "running": 0, "size": 1}}] 2022-03-10T03:22:12.593 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph orch ls -f json 2022-03-10T03:22:13.294 INFO:teuthology.orchestra.run.smithi037.stdout: 2022-03-10T03:22:13.294 INFO:teuthology.orchestra.run.smithi037.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:31.000000Z", "last_refresh": "2022-03-10T03:22:13.129398Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:11:53.000000Z", "last_refresh": "2022-03-10T03:22:13.129398Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:09:03.000000Z", "last_refresh": "2022-03-10T03:22:13.129398Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:30.000000Z", "last_refresh": "2022-03-10T03:22:13.129398Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T03:22:13.129398Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T03:22:13.129398Z", "running": 0, "size": 1}}] 2022-03-10T03:22:23.324 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph orch ls -f json 2022-03-10T03:22:24.062 INFO:teuthology.orchestra.run.smithi037.stdout: 2022-03-10T03:22:24.063 INFO:teuthology.orchestra.run.smithi037.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:31.000000Z", "last_refresh": "2022-03-10T03:22:23.883985Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:11:53.000000Z", "last_refresh": "2022-03-10T03:22:23.883985Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:09:03.000000Z", "last_refresh": "2022-03-10T03:22:23.883985Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:30.000000Z", "last_refresh": "2022-03-10T03:22:23.883985Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T03:22:23.883985Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T03:22:23.883985Z", "running": 0, "size": 1}}] 2022-03-10T03:22:34.092 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph orch ls -f json 2022-03-10T03:22:34.863 INFO:teuthology.orchestra.run.smithi037.stdout: 2022-03-10T03:22:34.863 INFO:teuthology.orchestra.run.smithi037.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:31.000000Z", "last_refresh": "2022-03-10T03:22:34.654711Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:11:53.000000Z", "last_refresh": "2022-03-10T03:22:34.654711Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:09:03.000000Z", "last_refresh": "2022-03-10T03:22:34.654711Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:30.000000Z", "last_refresh": "2022-03-10T03:22:34.654711Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T03:22:34.654711Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T03:22:34.654711Z", "running": 0, "size": 1}}] 2022-03-10T03:22:44.890 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph orch ls -f json 2022-03-10T03:22:45.623 INFO:teuthology.orchestra.run.smithi037.stdout: 2022-03-10T03:22:45.624 INFO:teuthology.orchestra.run.smithi037.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:31.000000Z", "last_refresh": "2022-03-10T03:22:45.444958Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:11:53.000000Z", "last_refresh": "2022-03-10T03:22:45.444958Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:09:03.000000Z", "last_refresh": "2022-03-10T03:22:45.444958Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:30.000000Z", "last_refresh": "2022-03-10T03:22:45.444958Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T03:22:45.444958Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T03:22:45.444958Z", "running": 0, "size": 1}}] 2022-03-10T03:22:55.647 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph orch ls -f json 2022-03-10T03:22:56.380 INFO:teuthology.orchestra.run.smithi037.stdout: 2022-03-10T03:22:56.380 INFO:teuthology.orchestra.run.smithi037.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:31.000000Z", "last_refresh": "2022-03-10T03:22:56.203166Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:11:53.000000Z", "last_refresh": "2022-03-10T03:22:56.203166Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:09:03.000000Z", "last_refresh": "2022-03-10T03:22:56.203166Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:30.000000Z", "last_refresh": "2022-03-10T03:22:56.203166Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T03:22:56.203166Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T03:22:56.203166Z", "running": 0, "size": 1}}] 2022-03-10T03:22:58.347 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:22:58.348946 I | op-mon: parsing mon endpoints: a=10.100.186.100:6789,b=10.111.132.151:6789,c=10.101.46.100:6789 2022-03-10T03:22:59.014 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:22:59.016336 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T03:22:59.005+0000 7f355bf34380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T03:22:59.015 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10T03:22:59.005+0000 7f355bf34380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T03:22:59.015 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10T03:22:59.005+0000 7f355bf34380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T03:22:59.015 INFO:tasks.rook.operator.smithi037.stdout:couldn't init storage provider. : exit status 5 2022-03-10T03:23:06.434 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph orch ls -f json 2022-03-10T03:23:07.203 INFO:teuthology.orchestra.run.smithi037.stdout: 2022-03-10T03:23:07.203 INFO:teuthology.orchestra.run.smithi037.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:31.000000Z", "last_refresh": "2022-03-10T03:23:07.004158Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:11:53.000000Z", "last_refresh": "2022-03-10T03:23:07.004158Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:09:03.000000Z", "last_refresh": "2022-03-10T03:23:07.004158Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:30.000000Z", "last_refresh": "2022-03-10T03:23:07.004158Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T03:23:07.004158Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T03:23:07.004158Z", "running": 0, "size": 1}}] 2022-03-10T03:23:17.229 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph orch ls -f json 2022-03-10T03:23:17.961 INFO:teuthology.orchestra.run.smithi037.stdout: 2022-03-10T03:23:17.962 INFO:teuthology.orchestra.run.smithi037.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:31.000000Z", "last_refresh": "2022-03-10T03:23:17.791248Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:11:53.000000Z", "last_refresh": "2022-03-10T03:23:17.791248Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:09:03.000000Z", "last_refresh": "2022-03-10T03:23:17.791248Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:30.000000Z", "last_refresh": "2022-03-10T03:23:17.791248Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T03:23:17.791248Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T03:23:17.791248Z", "running": 0, "size": 1}}] 2022-03-10T03:23:27.990 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph orch ls -f json 2022-03-10T03:23:28.711 INFO:teuthology.orchestra.run.smithi037.stdout: 2022-03-10T03:23:28.712 INFO:teuthology.orchestra.run.smithi037.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:31.000000Z", "last_refresh": "2022-03-10T03:23:28.536238Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:11:53.000000Z", "last_refresh": "2022-03-10T03:23:28.536238Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:09:03.000000Z", "last_refresh": "2022-03-10T03:23:28.536238Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:30.000000Z", "last_refresh": "2022-03-10T03:23:28.536238Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T03:23:28.536238Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T03:23:28.536238Z", "running": 0, "size": 1}}] 2022-03-10T03:23:38.738 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph orch ls -f json 2022-03-10T03:23:39.453 INFO:teuthology.orchestra.run.smithi037.stdout: 2022-03-10T03:23:39.453 INFO:teuthology.orchestra.run.smithi037.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:31.000000Z", "last_refresh": "2022-03-10T03:23:39.282200Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:11:53.000000Z", "last_refresh": "2022-03-10T03:23:39.282200Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:09:03.000000Z", "last_refresh": "2022-03-10T03:23:39.282200Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:30.000000Z", "last_refresh": "2022-03-10T03:23:39.282200Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T03:23:39.282200Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T03:23:39.282200Z", "running": 0, "size": 1}}] 2022-03-10T03:23:49.479 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph orch ls -f json 2022-03-10T03:23:50.163 INFO:teuthology.orchestra.run.smithi037.stdout: 2022-03-10T03:23:50.164 INFO:teuthology.orchestra.run.smithi037.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:31.000000Z", "last_refresh": "2022-03-10T03:23:50.000486Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:11:53.000000Z", "last_refresh": "2022-03-10T03:23:50.000486Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:09:03.000000Z", "last_refresh": "2022-03-10T03:23:50.000486Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:30.000000Z", "last_refresh": "2022-03-10T03:23:50.000486Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T03:23:50.000486Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T03:23:50.000486Z", "running": 0, "size": 1}}] 2022-03-10T03:24:00.188 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph orch ls -f json 2022-03-10T03:24:00.924 INFO:teuthology.orchestra.run.smithi037.stdout: 2022-03-10T03:24:00.925 INFO:teuthology.orchestra.run.smithi037.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:31.000000Z", "last_refresh": "2022-03-10T03:24:00.756559Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:11:53.000000Z", "last_refresh": "2022-03-10T03:24:00.756559Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:09:03.000000Z", "last_refresh": "2022-03-10T03:24:00.756559Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:30.000000Z", "last_refresh": "2022-03-10T03:24:00.756559Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T03:24:00.756559Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T03:24:00.756559Z", "running": 0, "size": 1}}] 2022-03-10T03:24:10.952 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph orch ls -f json 2022-03-10T03:24:11.677 INFO:teuthology.orchestra.run.smithi037.stdout: 2022-03-10T03:24:11.678 INFO:teuthology.orchestra.run.smithi037.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:31.000000Z", "last_refresh": "2022-03-10T03:24:11.506863Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:11:53.000000Z", "last_refresh": "2022-03-10T03:24:11.506863Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:09:03.000000Z", "last_refresh": "2022-03-10T03:24:11.506863Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:30.000000Z", "last_refresh": "2022-03-10T03:24:11.506863Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T03:24:11.506863Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T03:24:11.506863Z", "running": 0, "size": 1}}] 2022-03-10T03:24:21.706 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph orch ls -f json 2022-03-10T03:24:22.421 INFO:teuthology.orchestra.run.smithi037.stdout: 2022-03-10T03:24:22.421 INFO:teuthology.orchestra.run.smithi037.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:31.000000Z", "last_refresh": "2022-03-10T03:24:22.250242Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:11:53.000000Z", "last_refresh": "2022-03-10T03:24:22.250242Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:09:03.000000Z", "last_refresh": "2022-03-10T03:24:22.250242Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:30.000000Z", "last_refresh": "2022-03-10T03:24:22.250242Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T03:24:22.250242Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T03:24:22.250242Z", "running": 0, "size": 1}}] 2022-03-10T03:24:32.447 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph orch ls -f json 2022-03-10T03:24:33.233 INFO:teuthology.orchestra.run.smithi037.stdout: 2022-03-10T03:24:33.233 INFO:teuthology.orchestra.run.smithi037.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:31.000000Z", "last_refresh": "2022-03-10T03:24:33.007691Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:11:53.000000Z", "last_refresh": "2022-03-10T03:24:33.007691Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:09:03.000000Z", "last_refresh": "2022-03-10T03:24:33.007691Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:30.000000Z", "last_refresh": "2022-03-10T03:24:33.007691Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T03:24:33.007691Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T03:24:33.007691Z", "running": 0, "size": 1}}] 2022-03-10T03:24:43.259 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph orch ls -f json 2022-03-10T03:24:44.237 INFO:teuthology.orchestra.run.smithi037.stdout: 2022-03-10T03:24:44.238 INFO:teuthology.orchestra.run.smithi037.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:31.000000Z", "last_refresh": "2022-03-10T03:24:43.796314Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:11:53.000000Z", "last_refresh": "2022-03-10T03:24:43.796314Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:09:03.000000Z", "last_refresh": "2022-03-10T03:24:43.796314Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:30.000000Z", "last_refresh": "2022-03-10T03:24:43.796314Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T03:24:43.796314Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T03:24:43.796314Z", "running": 0, "size": 1}}] 2022-03-10T03:24:54.262 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph orch ls -f json 2022-03-10T03:24:54.998 INFO:teuthology.orchestra.run.smithi037.stdout: 2022-03-10T03:24:54.998 INFO:teuthology.orchestra.run.smithi037.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:31.000000Z", "last_refresh": "2022-03-10T03:24:54.798051Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:11:53.000000Z", "last_refresh": "2022-03-10T03:24:54.798051Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:09:03.000000Z", "last_refresh": "2022-03-10T03:24:54.798051Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:30.000000Z", "last_refresh": "2022-03-10T03:24:54.798051Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T03:24:54.798051Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T03:24:54.798051Z", "running": 0, "size": 1}}] 2022-03-10T03:25:05.021 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph orch ls -f json 2022-03-10T03:25:05.781 INFO:teuthology.orchestra.run.smithi037.stdout: 2022-03-10T03:25:05.782 INFO:teuthology.orchestra.run.smithi037.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:31.000000Z", "last_refresh": "2022-03-10T03:25:05.577393Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:11:53.000000Z", "last_refresh": "2022-03-10T03:25:05.577393Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:09:03.000000Z", "last_refresh": "2022-03-10T03:25:05.577393Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:30.000000Z", "last_refresh": "2022-03-10T03:25:05.577393Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T03:25:05.577393Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T03:25:05.577393Z", "running": 0, "size": 1}}] 2022-03-10T03:25:15.812 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph orch ls -f json 2022-03-10T03:25:16.575 INFO:teuthology.orchestra.run.smithi037.stdout: 2022-03-10T03:25:16.575 INFO:teuthology.orchestra.run.smithi037.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:31.000000Z", "last_refresh": "2022-03-10T03:25:16.413083Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:11:53.000000Z", "last_refresh": "2022-03-10T03:25:16.413083Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:09:03.000000Z", "last_refresh": "2022-03-10T03:25:16.413083Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:30.000000Z", "last_refresh": "2022-03-10T03:25:16.413083Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T03:25:16.413083Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T03:25:16.413083Z", "running": 0, "size": 1}}] 2022-03-10T03:25:26.609 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph orch ls -f json 2022-03-10T03:25:27.435 INFO:teuthology.orchestra.run.smithi037.stdout: 2022-03-10T03:25:27.435 INFO:teuthology.orchestra.run.smithi037.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:31.000000Z", "last_refresh": "2022-03-10T03:25:27.211195Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:11:53.000000Z", "last_refresh": "2022-03-10T03:25:27.211195Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:09:03.000000Z", "last_refresh": "2022-03-10T03:25:27.211195Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:30.000000Z", "last_refresh": "2022-03-10T03:25:27.211195Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T03:25:27.211195Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T03:25:27.211195Z", "running": 0, "size": 1}}] 2022-03-10T03:25:37.462 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph orch ls -f json 2022-03-10T03:25:38.213 INFO:teuthology.orchestra.run.smithi037.stdout: 2022-03-10T03:25:38.214 INFO:teuthology.orchestra.run.smithi037.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:31.000000Z", "last_refresh": "2022-03-10T03:25:38.037751Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:11:53.000000Z", "last_refresh": "2022-03-10T03:25:38.037751Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:09:03.000000Z", "last_refresh": "2022-03-10T03:25:38.037751Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:30.000000Z", "last_refresh": "2022-03-10T03:25:38.037751Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T03:25:38.037751Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T03:25:38.037751Z", "running": 0, "size": 1}}] 2022-03-10T03:25:48.262 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph orch ls -f json 2022-03-10T03:25:48.998 INFO:teuthology.orchestra.run.smithi037.stdout: 2022-03-10T03:25:48.998 INFO:teuthology.orchestra.run.smithi037.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:31.000000Z", "last_refresh": "2022-03-10T03:25:48.803977Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:11:53.000000Z", "last_refresh": "2022-03-10T03:25:48.803977Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:09:03.000000Z", "last_refresh": "2022-03-10T03:25:48.803977Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:30.000000Z", "last_refresh": "2022-03-10T03:25:48.803977Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T03:25:48.803977Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T03:25:48.803977Z", "running": 0, "size": 1}}] 2022-03-10T03:25:59.026 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph orch ls -f json 2022-03-10T03:25:59.825 INFO:teuthology.orchestra.run.smithi037.stdout: 2022-03-10T03:25:59.826 INFO:teuthology.orchestra.run.smithi037.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:31.000000Z", "last_refresh": "2022-03-10T03:25:59.608350Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:11:53.000000Z", "last_refresh": "2022-03-10T03:25:59.608350Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:09:03.000000Z", "last_refresh": "2022-03-10T03:25:59.608350Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:30.000000Z", "last_refresh": "2022-03-10T03:25:59.608350Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T03:25:59.608350Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T03:25:59.608350Z", "running": 0, "size": 1}}] 2022-03-10T03:26:09.853 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph orch ls -f json 2022-03-10T03:26:10.591 INFO:teuthology.orchestra.run.smithi037.stdout: 2022-03-10T03:26:10.591 INFO:teuthology.orchestra.run.smithi037.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:31.000000Z", "last_refresh": "2022-03-10T03:26:10.413158Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:11:53.000000Z", "last_refresh": "2022-03-10T03:26:10.413158Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:09:03.000000Z", "last_refresh": "2022-03-10T03:26:10.413158Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:30.000000Z", "last_refresh": "2022-03-10T03:26:10.413158Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T03:26:10.413158Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T03:26:10.413158Z", "running": 0, "size": 1}}] 2022-03-10T03:26:20.614 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph orch ls -f json 2022-03-10T03:26:21.357 INFO:teuthology.orchestra.run.smithi037.stdout: 2022-03-10T03:26:21.358 INFO:teuthology.orchestra.run.smithi037.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:31.000000Z", "last_refresh": "2022-03-10T03:26:21.194669Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:11:53.000000Z", "last_refresh": "2022-03-10T03:26:21.194669Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:09:03.000000Z", "last_refresh": "2022-03-10T03:26:21.194669Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:30.000000Z", "last_refresh": "2022-03-10T03:26:21.194669Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T03:26:21.194669Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T03:26:21.194669Z", "running": 0, "size": 1}}] 2022-03-10T03:26:31.384 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph orch ls -f json 2022-03-10T03:26:32.156 INFO:teuthology.orchestra.run.smithi037.stdout: 2022-03-10T03:26:32.156 INFO:teuthology.orchestra.run.smithi037.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:31.000000Z", "last_refresh": "2022-03-10T03:26:31.939616Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:11:53.000000Z", "last_refresh": "2022-03-10T03:26:31.939616Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:09:03.000000Z", "last_refresh": "2022-03-10T03:26:31.939616Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:30.000000Z", "last_refresh": "2022-03-10T03:26:31.939616Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T03:26:31.939616Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T03:26:31.939616Z", "running": 0, "size": 1}}] 2022-03-10T03:26:42.181 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph orch ls -f json 2022-03-10T03:26:42.920 INFO:teuthology.orchestra.run.smithi037.stdout: 2022-03-10T03:26:42.920 INFO:teuthology.orchestra.run.smithi037.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:31.000000Z", "last_refresh": "2022-03-10T03:26:42.722284Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:11:53.000000Z", "last_refresh": "2022-03-10T03:26:42.722284Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:09:03.000000Z", "last_refresh": "2022-03-10T03:26:42.722284Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:30.000000Z", "last_refresh": "2022-03-10T03:26:42.722284Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T03:26:42.722284Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T03:26:42.722284Z", "running": 0, "size": 1}}] 2022-03-10T03:26:52.946 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph orch ls -f json 2022-03-10T03:26:53.731 INFO:teuthology.orchestra.run.smithi037.stdout: 2022-03-10T03:26:53.731 INFO:teuthology.orchestra.run.smithi037.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:31.000000Z", "last_refresh": "2022-03-10T03:26:53.508359Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:11:53.000000Z", "last_refresh": "2022-03-10T03:26:53.508359Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:09:03.000000Z", "last_refresh": "2022-03-10T03:26:53.508359Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:30.000000Z", "last_refresh": "2022-03-10T03:26:53.508359Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T03:26:53.508359Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T03:26:53.508359Z", "running": 0, "size": 1}}] 2022-03-10T03:27:03.756 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph orch ls -f json 2022-03-10T03:27:04.477 INFO:teuthology.orchestra.run.smithi037.stdout: 2022-03-10T03:27:04.477 INFO:teuthology.orchestra.run.smithi037.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:31.000000Z", "last_refresh": "2022-03-10T03:27:04.305653Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:11:53.000000Z", "last_refresh": "2022-03-10T03:27:04.305653Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:09:03.000000Z", "last_refresh": "2022-03-10T03:27:04.305653Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:30.000000Z", "last_refresh": "2022-03-10T03:27:04.305653Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T03:27:04.305653Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T03:27:04.305653Z", "running": 0, "size": 1}}] 2022-03-10T03:27:14.516 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph orch ls -f json 2022-03-10T03:27:15.229 INFO:teuthology.orchestra.run.smithi037.stdout: 2022-03-10T03:27:15.229 INFO:teuthology.orchestra.run.smithi037.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:31.000000Z", "last_refresh": "2022-03-10T03:27:15.073206Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:11:53.000000Z", "last_refresh": "2022-03-10T03:27:15.073206Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:09:03.000000Z", "last_refresh": "2022-03-10T03:27:15.073206Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:30.000000Z", "last_refresh": "2022-03-10T03:27:15.073206Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T03:27:15.073206Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T03:27:15.073206Z", "running": 0, "size": 1}}] 2022-03-10T03:27:25.257 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph orch ls -f json 2022-03-10T03:27:28.396 INFO:teuthology.orchestra.run.smithi037.stdout: 2022-03-10T03:27:28.396 INFO:teuthology.orchestra.run.smithi037.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:31.000000Z", "last_refresh": "2022-03-10T03:27:28.208986Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:11:53.000000Z", "last_refresh": "2022-03-10T03:27:28.208986Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:09:03.000000Z", "last_refresh": "2022-03-10T03:27:28.208986Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:30.000000Z", "last_refresh": "2022-03-10T03:27:28.208986Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T03:27:28.208986Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T03:27:28.208986Z", "running": 0, "size": 1}}] 2022-03-10T03:27:38.424 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph orch ls -f json 2022-03-10T03:27:39.130 INFO:teuthology.orchestra.run.smithi037.stdout: 2022-03-10T03:27:39.130 INFO:teuthology.orchestra.run.smithi037.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:31.000000Z", "last_refresh": "2022-03-10T03:27:38.964310Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:11:53.000000Z", "last_refresh": "2022-03-10T03:27:38.964310Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:09:03.000000Z", "last_refresh": "2022-03-10T03:27:38.964310Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:30.000000Z", "last_refresh": "2022-03-10T03:27:38.964310Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T03:27:38.964310Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T03:27:38.964310Z", "running": 0, "size": 1}}] 2022-03-10T03:27:49.157 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph orch ls -f json 2022-03-10T03:27:49.899 INFO:teuthology.orchestra.run.smithi037.stdout: 2022-03-10T03:27:49.900 INFO:teuthology.orchestra.run.smithi037.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:31.000000Z", "last_refresh": "2022-03-10T03:27:49.731540Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:11:53.000000Z", "last_refresh": "2022-03-10T03:27:49.731540Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:09:03.000000Z", "last_refresh": "2022-03-10T03:27:49.731540Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:30.000000Z", "last_refresh": "2022-03-10T03:27:49.731540Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T03:27:49.731540Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T03:27:49.731540Z", "running": 0, "size": 1}}] 2022-03-10T03:27:59.926 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph orch ls -f json 2022-03-10T03:28:00.716 INFO:teuthology.orchestra.run.smithi037.stdout: 2022-03-10T03:28:00.716 INFO:teuthology.orchestra.run.smithi037.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:31.000000Z", "last_refresh": "2022-03-10T03:28:00.483884Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:11:53.000000Z", "last_refresh": "2022-03-10T03:28:00.483884Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:09:03.000000Z", "last_refresh": "2022-03-10T03:28:00.483884Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:30.000000Z", "last_refresh": "2022-03-10T03:28:00.483884Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T03:28:00.483884Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T03:28:00.483884Z", "running": 0, "size": 1}}] 2022-03-10T03:28:10.744 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph orch ls -f json 2022-03-10T03:28:11.447 INFO:teuthology.orchestra.run.smithi037.stdout: 2022-03-10T03:28:11.447 INFO:teuthology.orchestra.run.smithi037.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:31.000000Z", "last_refresh": "2022-03-10T03:28:11.286204Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:11:53.000000Z", "last_refresh": "2022-03-10T03:28:11.286204Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:09:03.000000Z", "last_refresh": "2022-03-10T03:28:11.286204Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:30.000000Z", "last_refresh": "2022-03-10T03:28:11.286204Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T03:28:11.286204Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T03:28:11.286204Z", "running": 0, "size": 1}}] 2022-03-10T03:28:21.468 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph orch ls -f json 2022-03-10T03:28:22.179 INFO:teuthology.orchestra.run.smithi037.stdout: 2022-03-10T03:28:22.179 INFO:teuthology.orchestra.run.smithi037.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:31.000000Z", "last_refresh": "2022-03-10T03:28:21.993407Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:11:53.000000Z", "last_refresh": "2022-03-10T03:28:21.993407Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:09:03.000000Z", "last_refresh": "2022-03-10T03:28:21.993407Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:30.000000Z", "last_refresh": "2022-03-10T03:28:21.993407Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T03:28:21.993407Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T03:28:21.993407Z", "running": 0, "size": 1}}] 2022-03-10T03:28:26.722 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:28:26.723990 I | op-mon: parsing mon endpoints: a=10.100.186.100:6789,b=10.111.132.151:6789,c=10.101.46.100:6789 2022-03-10T03:28:27.432 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:28:27.434322 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T03:28:27.423+0000 7fbc26996380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T03:28:27.433 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10T03:28:27.423+0000 7fbc26996380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T03:28:27.433 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10T03:28:27.423+0000 7fbc26996380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T03:28:27.433 INFO:tasks.rook.operator.smithi037.stdout:couldn't init storage provider. : exit status 5 2022-03-10T03:28:32.203 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph orch ls -f json 2022-03-10T03:28:32.982 INFO:teuthology.orchestra.run.smithi037.stdout: 2022-03-10T03:28:32.983 INFO:teuthology.orchestra.run.smithi037.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:31.000000Z", "last_refresh": "2022-03-10T03:28:32.770924Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:11:53.000000Z", "last_refresh": "2022-03-10T03:28:32.770924Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:09:03.000000Z", "last_refresh": "2022-03-10T03:28:32.770924Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:30.000000Z", "last_refresh": "2022-03-10T03:28:32.770924Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T03:28:32.770924Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T03:28:32.770924Z", "running": 0, "size": 1}}] 2022-03-10T03:28:43.011 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph orch ls -f json 2022-03-10T03:28:43.754 INFO:teuthology.orchestra.run.smithi037.stdout: 2022-03-10T03:28:43.755 INFO:teuthology.orchestra.run.smithi037.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:31.000000Z", "last_refresh": "2022-03-10T03:28:43.590056Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:11:53.000000Z", "last_refresh": "2022-03-10T03:28:43.590056Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:09:03.000000Z", "last_refresh": "2022-03-10T03:28:43.590056Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:30.000000Z", "last_refresh": "2022-03-10T03:28:43.590056Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T03:28:43.590056Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T03:28:43.590056Z", "running": 0, "size": 1}}] 2022-03-10T03:28:53.781 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph orch ls -f json 2022-03-10T03:28:54.566 INFO:teuthology.orchestra.run.smithi037.stdout: 2022-03-10T03:28:54.567 INFO:teuthology.orchestra.run.smithi037.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:31.000000Z", "last_refresh": "2022-03-10T03:28:54.368899Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:11:53.000000Z", "last_refresh": "2022-03-10T03:28:54.368899Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:09:03.000000Z", "last_refresh": "2022-03-10T03:28:54.368899Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:30.000000Z", "last_refresh": "2022-03-10T03:28:54.368899Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T03:28:54.368899Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T03:28:54.368899Z", "running": 0, "size": 1}}] 2022-03-10T03:29:04.594 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph orch ls -f json 2022-03-10T03:29:05.285 INFO:teuthology.orchestra.run.smithi037.stdout: 2022-03-10T03:29:05.285 INFO:teuthology.orchestra.run.smithi037.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:31.000000Z", "last_refresh": "2022-03-10T03:29:05.114161Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:11:53.000000Z", "last_refresh": "2022-03-10T03:29:05.114161Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:09:03.000000Z", "last_refresh": "2022-03-10T03:29:05.114161Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:30.000000Z", "last_refresh": "2022-03-10T03:29:05.114161Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T03:29:05.114161Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T03:29:05.114161Z", "running": 0, "size": 1}}] 2022-03-10T03:29:15.312 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph orch ls -f json 2022-03-10T03:29:16.006 INFO:teuthology.orchestra.run.smithi037.stdout: 2022-03-10T03:29:16.006 INFO:teuthology.orchestra.run.smithi037.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:31.000000Z", "last_refresh": "2022-03-10T03:29:15.847845Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:11:53.000000Z", "last_refresh": "2022-03-10T03:29:15.847845Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:09:03.000000Z", "last_refresh": "2022-03-10T03:29:15.847845Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:30.000000Z", "last_refresh": "2022-03-10T03:29:15.847845Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T03:29:15.847845Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T03:29:15.847845Z", "running": 0, "size": 1}}] 2022-03-10T03:29:26.033 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph orch ls -f json 2022-03-10T03:29:26.823 INFO:teuthology.orchestra.run.smithi037.stdout: 2022-03-10T03:29:26.824 INFO:teuthology.orchestra.run.smithi037.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:31.000000Z", "last_refresh": "2022-03-10T03:29:26.656087Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:11:53.000000Z", "last_refresh": "2022-03-10T03:29:26.656087Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:09:03.000000Z", "last_refresh": "2022-03-10T03:29:26.656087Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:30.000000Z", "last_refresh": "2022-03-10T03:29:26.656087Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T03:29:26.656087Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T03:29:26.656087Z", "running": 0, "size": 1}}] 2022-03-10T03:29:36.852 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph orch ls -f json 2022-03-10T03:29:37.634 INFO:teuthology.orchestra.run.smithi037.stdout: 2022-03-10T03:29:37.634 INFO:teuthology.orchestra.run.smithi037.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:31.000000Z", "last_refresh": "2022-03-10T03:29:37.461354Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:11:53.000000Z", "last_refresh": "2022-03-10T03:29:37.461354Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:09:03.000000Z", "last_refresh": "2022-03-10T03:29:37.461354Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:30.000000Z", "last_refresh": "2022-03-10T03:29:37.461354Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T03:29:37.461354Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T03:29:37.461354Z", "running": 0, "size": 1}}] 2022-03-10T03:29:47.661 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph orch ls -f json 2022-03-10T03:29:48.492 INFO:teuthology.orchestra.run.smithi037.stdout: 2022-03-10T03:29:48.493 INFO:teuthology.orchestra.run.smithi037.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:31.000000Z", "last_refresh": "2022-03-10T03:29:48.190822Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:11:53.000000Z", "last_refresh": "2022-03-10T03:29:48.190822Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:09:03.000000Z", "last_refresh": "2022-03-10T03:29:48.190822Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:30.000000Z", "last_refresh": "2022-03-10T03:29:48.190822Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T03:29:48.190822Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T03:29:48.190822Z", "running": 0, "size": 1}}] 2022-03-10T03:29:58.518 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph orch ls -f json 2022-03-10T03:29:59.298 INFO:teuthology.orchestra.run.smithi037.stdout: 2022-03-10T03:29:59.299 INFO:teuthology.orchestra.run.smithi037.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:31.000000Z", "last_refresh": "2022-03-10T03:29:59.112705Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:11:53.000000Z", "last_refresh": "2022-03-10T03:29:59.112705Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:09:03.000000Z", "last_refresh": "2022-03-10T03:29:59.112705Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:30.000000Z", "last_refresh": "2022-03-10T03:29:59.112705Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T03:29:59.112705Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T03:29:59.112705Z", "running": 0, "size": 1}}] 2022-03-10T03:30:09.330 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph orch ls -f json 2022-03-10T03:30:10.081 INFO:teuthology.orchestra.run.smithi037.stdout: 2022-03-10T03:30:10.082 INFO:teuthology.orchestra.run.smithi037.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:31.000000Z", "last_refresh": "2022-03-10T03:30:09.900611Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:11:53.000000Z", "last_refresh": "2022-03-10T03:30:09.900611Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:09:03.000000Z", "last_refresh": "2022-03-10T03:30:09.900611Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:30.000000Z", "last_refresh": "2022-03-10T03:30:09.900611Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T03:30:09.900611Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T03:30:09.900611Z", "running": 0, "size": 1}}] 2022-03-10T03:30:20.109 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph orch ls -f json 2022-03-10T03:30:21.221 INFO:teuthology.orchestra.run.smithi037.stdout: 2022-03-10T03:30:21.221 INFO:teuthology.orchestra.run.smithi037.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:31.000000Z", "last_refresh": "2022-03-10T03:30:20.673776Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:11:53.000000Z", "last_refresh": "2022-03-10T03:30:20.673776Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:09:03.000000Z", "last_refresh": "2022-03-10T03:30:20.673776Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:30.000000Z", "last_refresh": "2022-03-10T03:30:20.673776Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T03:30:20.673776Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T03:30:20.673776Z", "running": 0, "size": 1}}] 2022-03-10T03:30:31.250 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph orch ls -f json 2022-03-10T03:30:31.998 INFO:teuthology.orchestra.run.smithi037.stdout: 2022-03-10T03:30:31.998 INFO:teuthology.orchestra.run.smithi037.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:31.000000Z", "last_refresh": "2022-03-10T03:30:31.781157Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:11:53.000000Z", "last_refresh": "2022-03-10T03:30:31.781157Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:09:03.000000Z", "last_refresh": "2022-03-10T03:30:31.781157Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:30.000000Z", "last_refresh": "2022-03-10T03:30:31.781157Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T03:30:31.781157Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T03:30:31.781157Z", "running": 0, "size": 1}}] 2022-03-10T03:30:42.026 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph orch ls -f json 2022-03-10T03:30:42.721 INFO:teuthology.orchestra.run.smithi037.stdout: 2022-03-10T03:30:42.722 INFO:teuthology.orchestra.run.smithi037.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:31.000000Z", "last_refresh": "2022-03-10T03:30:42.554004Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:11:53.000000Z", "last_refresh": "2022-03-10T03:30:42.554004Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:09:03.000000Z", "last_refresh": "2022-03-10T03:30:42.554004Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:30.000000Z", "last_refresh": "2022-03-10T03:30:42.554004Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T03:30:42.554004Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T03:30:42.554004Z", "running": 0, "size": 1}}] 2022-03-10T03:30:52.748 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph orch ls -f json 2022-03-10T03:30:53.473 INFO:teuthology.orchestra.run.smithi037.stdout: 2022-03-10T03:30:53.474 INFO:teuthology.orchestra.run.smithi037.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:31.000000Z", "last_refresh": "2022-03-10T03:30:53.286259Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:11:53.000000Z", "last_refresh": "2022-03-10T03:30:53.286259Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:09:03.000000Z", "last_refresh": "2022-03-10T03:30:53.286259Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:30.000000Z", "last_refresh": "2022-03-10T03:30:53.286259Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T03:30:53.286259Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T03:30:53.286259Z", "running": 0, "size": 1}}] 2022-03-10T03:31:03.501 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph orch ls -f json 2022-03-10T03:31:04.271 INFO:teuthology.orchestra.run.smithi037.stdout: 2022-03-10T03:31:04.271 INFO:teuthology.orchestra.run.smithi037.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:31.000000Z", "last_refresh": "2022-03-10T03:31:04.101944Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:11:53.000000Z", "last_refresh": "2022-03-10T03:31:04.101944Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:09:03.000000Z", "last_refresh": "2022-03-10T03:31:04.101944Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:30.000000Z", "last_refresh": "2022-03-10T03:31:04.101944Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T03:31:04.101944Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T03:31:04.101944Z", "running": 0, "size": 1}}] 2022-03-10T03:31:14.305 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph orch ls -f json 2022-03-10T03:31:15.036 INFO:teuthology.orchestra.run.smithi037.stdout: 2022-03-10T03:31:15.036 INFO:teuthology.orchestra.run.smithi037.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:31.000000Z", "last_refresh": "2022-03-10T03:31:14.854496Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:11:53.000000Z", "last_refresh": "2022-03-10T03:31:14.854496Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:09:03.000000Z", "last_refresh": "2022-03-10T03:31:14.854496Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:30.000000Z", "last_refresh": "2022-03-10T03:31:14.854496Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T03:31:14.854496Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T03:31:14.854496Z", "running": 0, "size": 1}}] 2022-03-10T03:31:25.058 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph orch ls -f json 2022-03-10T03:31:25.787 INFO:teuthology.orchestra.run.smithi037.stdout: 2022-03-10T03:31:25.788 INFO:teuthology.orchestra.run.smithi037.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:31.000000Z", "last_refresh": "2022-03-10T03:31:25.594949Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:11:53.000000Z", "last_refresh": "2022-03-10T03:31:25.594949Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:09:03.000000Z", "last_refresh": "2022-03-10T03:31:25.594949Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:30.000000Z", "last_refresh": "2022-03-10T03:31:25.594949Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T03:31:25.594949Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T03:31:25.594949Z", "running": 0, "size": 1}}] 2022-03-10T03:31:35.814 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph orch ls -f json 2022-03-10T03:31:36.575 INFO:teuthology.orchestra.run.smithi037.stdout: 2022-03-10T03:31:36.579 INFO:teuthology.orchestra.run.smithi037.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:31.000000Z", "last_refresh": "2022-03-10T03:31:36.401919Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:11:53.000000Z", "last_refresh": "2022-03-10T03:31:36.401919Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:09:03.000000Z", "last_refresh": "2022-03-10T03:31:36.401919Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:30.000000Z", "last_refresh": "2022-03-10T03:31:36.401919Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T03:31:36.401919Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T03:31:36.401919Z", "running": 0, "size": 1}}] 2022-03-10T03:31:46.604 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph orch ls -f json 2022-03-10T03:31:47.351 INFO:teuthology.orchestra.run.smithi037.stdout: 2022-03-10T03:31:47.352 INFO:teuthology.orchestra.run.smithi037.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:31.000000Z", "last_refresh": "2022-03-10T03:31:47.157798Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:11:53.000000Z", "last_refresh": "2022-03-10T03:31:47.157798Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:09:03.000000Z", "last_refresh": "2022-03-10T03:31:47.157798Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:30.000000Z", "last_refresh": "2022-03-10T03:31:47.157798Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T03:31:47.157798Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T03:31:47.157798Z", "running": 0, "size": 1}}] 2022-03-10T03:31:57.380 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph orch ls -f json 2022-03-10T03:31:58.096 INFO:teuthology.orchestra.run.smithi037.stdout: 2022-03-10T03:31:58.097 INFO:teuthology.orchestra.run.smithi037.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:31.000000Z", "last_refresh": "2022-03-10T03:31:57.922202Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:11:53.000000Z", "last_refresh": "2022-03-10T03:31:57.922202Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:09:03.000000Z", "last_refresh": "2022-03-10T03:31:57.922202Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:30.000000Z", "last_refresh": "2022-03-10T03:31:57.922202Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T03:31:57.922202Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T03:31:57.922202Z", "running": 0, "size": 1}}] 2022-03-10T03:32:08.123 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph orch ls -f json 2022-03-10T03:32:08.834 INFO:teuthology.orchestra.run.smithi037.stdout: 2022-03-10T03:32:08.834 INFO:teuthology.orchestra.run.smithi037.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:31.000000Z", "last_refresh": "2022-03-10T03:32:08.642015Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:11:53.000000Z", "last_refresh": "2022-03-10T03:32:08.642015Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:09:03.000000Z", "last_refresh": "2022-03-10T03:32:08.642015Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:30.000000Z", "last_refresh": "2022-03-10T03:32:08.642015Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T03:32:08.642015Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T03:32:08.642015Z", "running": 0, "size": 1}}] 2022-03-10T03:32:18.861 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph orch ls -f json 2022-03-10T03:32:19.650 INFO:teuthology.orchestra.run.smithi037.stdout: 2022-03-10T03:32:19.651 INFO:teuthology.orchestra.run.smithi037.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:31.000000Z", "last_refresh": "2022-03-10T03:32:19.475622Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:11:53.000000Z", "last_refresh": "2022-03-10T03:32:19.475622Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:09:03.000000Z", "last_refresh": "2022-03-10T03:32:19.475622Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:30.000000Z", "last_refresh": "2022-03-10T03:32:19.475622Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T03:32:19.475622Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T03:32:19.475622Z", "running": 0, "size": 1}}] 2022-03-10T03:32:29.677 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph orch ls -f json 2022-03-10T03:32:30.420 INFO:teuthology.orchestra.run.smithi037.stdout: 2022-03-10T03:32:30.420 INFO:teuthology.orchestra.run.smithi037.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:31.000000Z", "last_refresh": "2022-03-10T03:32:30.228873Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:11:53.000000Z", "last_refresh": "2022-03-10T03:32:30.228873Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:09:03.000000Z", "last_refresh": "2022-03-10T03:32:30.228873Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:30.000000Z", "last_refresh": "2022-03-10T03:32:30.228873Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T03:32:30.228873Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T03:32:30.228873Z", "running": 0, "size": 1}}] 2022-03-10T03:32:40.448 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph orch ls -f json 2022-03-10T03:32:41.148 INFO:teuthology.orchestra.run.smithi037.stdout: 2022-03-10T03:32:41.149 INFO:teuthology.orchestra.run.smithi037.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:31.000000Z", "last_refresh": "2022-03-10T03:32:40.979450Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:11:53.000000Z", "last_refresh": "2022-03-10T03:32:40.979450Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:09:03.000000Z", "last_refresh": "2022-03-10T03:32:40.979450Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:30.000000Z", "last_refresh": "2022-03-10T03:32:40.979450Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T03:32:40.979450Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T03:32:40.979450Z", "running": 0, "size": 1}}] 2022-03-10T03:32:51.172 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph orch ls -f json 2022-03-10T03:32:51.898 INFO:teuthology.orchestra.run.smithi037.stdout: 2022-03-10T03:32:51.898 INFO:teuthology.orchestra.run.smithi037.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:31.000000Z", "last_refresh": "2022-03-10T03:32:51.723141Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:11:53.000000Z", "last_refresh": "2022-03-10T03:32:51.723141Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:09:03.000000Z", "last_refresh": "2022-03-10T03:32:51.723141Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:30.000000Z", "last_refresh": "2022-03-10T03:32:51.723141Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T03:32:51.723141Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T03:32:51.723141Z", "running": 0, "size": 1}}] 2022-03-10T03:33:01.924 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph orch ls -f json 2022-03-10T03:33:02.652 INFO:teuthology.orchestra.run.smithi037.stdout: 2022-03-10T03:33:02.652 INFO:teuthology.orchestra.run.smithi037.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:31.000000Z", "last_refresh": "2022-03-10T03:33:02.461245Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:11:53.000000Z", "last_refresh": "2022-03-10T03:33:02.461245Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:09:03.000000Z", "last_refresh": "2022-03-10T03:33:02.461245Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:30.000000Z", "last_refresh": "2022-03-10T03:33:02.461245Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T03:33:02.461245Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T03:33:02.461245Z", "running": 0, "size": 1}}] 2022-03-10T03:33:12.679 DEBUG:teuthology.orchestra.run.smithi037:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-sj4qk -- ceph orch ls -f json 2022-03-10T03:33:13.554 INFO:teuthology.orchestra.run.smithi037.stdout: 2022-03-10T03:33:13.555 INFO:teuthology.orchestra.run.smithi037.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:31.000000Z", "last_refresh": "2022-03-10T03:33:13.233121Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:11:53.000000Z", "last_refresh": "2022-03-10T03:33:13.233121Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:09:03.000000Z", "last_refresh": "2022-03-10T03:33:13.233121Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T03:13:30.000000Z", "last_refresh": "2022-03-10T03:33:13.233121Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T03:33:13.233121Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T03:33:13.233121Z", "running": 0, "size": 1}}] 2022-03-10T03:33:13.582 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_ff5db9169566776b2f41faf671316ca58a318ea6/teuthology/contextutil.py", line 33, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_7a44e9ebca556862e9704be295a07719d7db47b3/qa/tasks/rook.py", line 669, in task while proceed(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_ff5db9169566776b2f41faf671316ca58a318ea6/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds 2022-03-10T03:33:13.583 ERROR:tasks.rook:'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_7a44e9ebca556862e9704be295a07719d7db47b3/qa/tasks/rook.py", line 530, in ceph_config_keyring yield File "/home/teuthworker/src/git.ceph.com_git_teuthology_ff5db9169566776b2f41faf671316ca58a318ea6/teuthology/contextutil.py", line 33, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_7a44e9ebca556862e9704be295a07719d7db47b3/qa/tasks/rook.py", line 669, in task while proceed(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_ff5db9169566776b2f41faf671316ca58a318ea6/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds 2022-03-10T03:33:13.584 INFO:tasks.rook:Cleaning up config and client.admin keyring 2022-03-10T03:33:13.584 DEBUG:teuthology.orchestra.run.smithi037:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2022-03-10T03:33:13.599 DEBUG:teuthology.orchestra.run.smithi038:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2022-03-10T03:33:13.630 DEBUG:teuthology.orchestra.run.smithi195:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2022-03-10T03:33:13.648 ERROR:tasks.rook:'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_7a44e9ebca556862e9704be295a07719d7db47b3/qa/tasks/rook.py", line 478, in rook_post_config yield File "/home/teuthworker/src/git.ceph.com_git_teuthology_ff5db9169566776b2f41faf671316ca58a318ea6/teuthology/contextutil.py", line 33, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_7a44e9ebca556862e9704be295a07719d7db47b3/qa/tasks/rook.py", line 669, in task while proceed(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_ff5db9169566776b2f41faf671316ca58a318ea6/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds 2022-03-10T03:33:13.648 ERROR:tasks.rook:'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_7a44e9ebca556862e9704be295a07719d7db47b3/qa/tasks/rook.py", line 442, in rook_toolbox yield File "/home/teuthworker/src/git.ceph.com_git_teuthology_ff5db9169566776b2f41faf671316ca58a318ea6/teuthology/contextutil.py", line 33, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_7a44e9ebca556862e9704be295a07719d7db47b3/qa/tasks/rook.py", line 669, in task while proceed(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_ff5db9169566776b2f41faf671316ca58a318ea6/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds 2022-03-10T03:33:13.661 DEBUG:teuthology.orchestra.remote:smithi037:rook/deploy/examples/operator.yaml is 20KB 2022-03-10T03:33:13.676 DEBUG:teuthology.orchestra.run.smithi037:> kubectl delete -f rook/deploy/examples/toolbox.yaml 2022-03-10T03:33:13.767 INFO:teuthology.orchestra.run.smithi037.stdout:deployment.apps "rook-ceph-tools" deleted 2022-03-10T03:33:13.801 ERROR:tasks.rook:'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_7a44e9ebca556862e9704be295a07719d7db47b3/qa/tasks/rook.py", line 379, in rook_cluster yield File "/home/teuthworker/src/git.ceph.com_git_teuthology_ff5db9169566776b2f41faf671316ca58a318ea6/teuthology/contextutil.py", line 33, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_7a44e9ebca556862e9704be295a07719d7db47b3/qa/tasks/rook.py", line 669, in task while proceed(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_ff5db9169566776b2f41faf671316ca58a318ea6/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds 2022-03-10T03:33:13.802 DEBUG:teuthology.orchestra.run.smithi037:> kubectl delete -f cluster.yaml 2022-03-10T03:33:13.865 INFO:teuthology.orchestra.run.smithi037.stdout:cephcluster.ceph.rook.io "rook-ceph" deleted 2022-03-10T03:33:13.870 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:33:13.872106 I | ceph-cluster-controller: CR "rook-ceph" is going be deleted, cancelling any ongoing orchestration 2022-03-10T03:33:13.870 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:33:13.872238 I | operator: reloading operator's CRDs manager, cancelling all orchestrations! 2022-03-10T03:33:13.875 INFO:tasks.rook.operator.smithi037.stdout:I0310 03:33:13.872660 1 manager.go:148] objectbucket.io/provisioner-manager "msg"="stopping provisioner" "name"="rook-ceph.ceph.rook.io/bucket" "reason"={} 2022-03-10T03:33:13.875 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:33:13.872694 I | operator: successfully started the controller-runtime manager 2022-03-10T03:33:13.916 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:33:13.918618 I | op-k8sutil: ROOK_CURRENT_NAMESPACE_ONLY="false" (env var) 2022-03-10T03:33:13.916 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:33:13.918654 I | operator: watching all namespaces for Ceph CRs 2022-03-10T03:33:13.917 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:33:13.918695 I | operator: setting up schemes 2022-03-10T03:33:13.920 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:33:13.922480 I | operator: setting up the controller-runtime manager 2022-03-10T03:33:14.225 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:33:14.227136 I | operator: looking for admission webhook secret "rook-ceph-admission-controller" 2022-03-10T03:33:14.230 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:33:14.229478 I | operator: admission webhook secret "rook-ceph-admission-controller" not found. proceeding without the admission controller 2022-03-10T03:33:14.231 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:33:14.229518 I | ceph-cluster-controller: successfully started 2022-03-10T03:33:14.231 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:33:14.229574 I | ceph-cluster-controller: enabling hotplug orchestration 2022-03-10T03:33:14.231 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:33:14.229599 I | ceph-crashcollector-controller: successfully started 2022-03-10T03:33:14.231 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:33:14.229641 I | ceph-block-pool-controller: successfully started 2022-03-10T03:33:14.232 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:33:14.229689 I | ceph-object-store-user-controller: successfully started 2022-03-10T03:33:14.232 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:33:14.229730 I | ceph-object-realm-controller: successfully started 2022-03-10T03:33:14.232 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:33:14.229750 I | ceph-object-zonegroup-controller: successfully started 2022-03-10T03:33:14.233 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:33:14.229767 I | ceph-object-zone-controller: successfully started 2022-03-10T03:33:14.233 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:33:14.229927 I | ceph-object-controller: successfully started 2022-03-10T03:33:14.234 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:33:14.229987 I | ceph-file-controller: successfully started 2022-03-10T03:33:14.234 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:33:14.230027 I | ceph-nfs-controller: successfully started 2022-03-10T03:33:14.234 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:33:14.230062 I | ceph-rbd-mirror-controller: successfully started 2022-03-10T03:33:14.234 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:33:14.230092 I | ceph-client-controller: successfully started 2022-03-10T03:33:14.235 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:33:14.230115 I | ceph-filesystem-mirror-controller: successfully started 2022-03-10T03:33:14.235 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:33:14.230146 I | operator: rook-ceph-operator-config-controller successfully started 2022-03-10T03:33:14.235 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:33:14.230168 I | ceph-csi: rook-ceph-operator-csi-controller successfully started 2022-03-10T03:33:14.235 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:33:14.230190 I | op-bucket-prov: rook-ceph-operator-bucket-controller successfully started 2022-03-10T03:33:14.236 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:33:14.230217 I | ceph-bucket-topic: successfully started 2022-03-10T03:33:14.236 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:33:14.230236 I | ceph-bucket-notification: successfully started 2022-03-10T03:33:14.236 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:33:14.230253 I | ceph-bucket-notification: successfully started 2022-03-10T03:33:14.237 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:33:14.230267 I | ceph-fs-subvolumegroup-controller: successfully started 2022-03-10T03:33:14.237 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:33:14.231789 I | operator: starting the controller-runtime manager 2022-03-10T03:33:14.335 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:33:14.337719 I | op-k8sutil: ROOK_CEPH_COMMANDS_TIMEOUT_SECONDS="15" (configmap) 2022-03-10T03:33:14.336 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:33:14.337785 I | op-k8sutil: ROOK_LOG_LEVEL="INFO" (configmap) 2022-03-10T03:33:14.336 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:33:14.337806 I | op-k8sutil: ROOK_ENABLE_DISCOVERY_DAEMON="false" (configmap) 2022-03-10T03:33:14.358 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:33:14.360565 I | operator: rook-ceph-operator-config-controller done reconciling 2022-03-10T03:33:14.395 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:33:14.397251 I | op-mon: parsing mon endpoints: a=10.100.186.100:6789,b=10.111.132.151:6789,c=10.101.46.100:6789 2022-03-10T03:33:14.425 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:33:14.427623 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:33:14.437 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:33:14.440260 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:33:15.081 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:33:15.082715 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T03:33:15.074+0000 7f26dd637380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T03:33:15.081 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10T03:33:15.074+0000 7f26dd637380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T03:33:15.081 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10T03:33:15.074+0000 7f26dd637380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T03:33:15.082 INFO:tasks.rook.operator.smithi037.stdout:couldn't init storage provider. : exit status 5 2022-03-10T03:33:15.135 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:33:15.138011 I | op-mon: parsing mon endpoints: a=10.100.186.100:6789,b=10.111.132.151:6789,c=10.101.46.100:6789 2022-03-10T03:33:15.809 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:33:15.810265 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T03:33:15.798+0000 7fc481b14380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T03:33:15.809 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10T03:33:15.798+0000 7fc481b14380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T03:33:15.809 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10T03:33:15.798+0000 7fc481b14380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T03:33:15.810 INFO:tasks.rook.operator.smithi037.stdout:couldn't init storage provider. : exit status 5 2022-03-10T03:33:15.860 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:33:15.862236 I | op-mon: parsing mon endpoints: a=10.100.186.100:6789,b=10.111.132.151:6789,c=10.101.46.100:6789 2022-03-10T03:33:16.547 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:33:16.548913 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T03:33:16.538+0000 7efdceb65380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T03:33:16.547 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10T03:33:16.538+0000 7efdceb65380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T03:33:16.547 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10T03:33:16.538+0000 7efdceb65380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T03:33:16.548 INFO:tasks.rook.operator.smithi037.stdout:couldn't init storage provider. : exit status 5 2022-03-10T03:33:16.655 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:33:16.657628 I | op-mon: parsing mon endpoints: a=10.100.186.100:6789,b=10.111.132.151:6789,c=10.101.46.100:6789 2022-03-10T03:33:17.362 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:33:17.364276 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T03:33:17.354+0000 7f164152d380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T03:33:17.363 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10T03:33:17.354+0000 7f164152d380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T03:33:17.363 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10T03:33:17.354+0000 7f164152d380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T03:33:17.363 INFO:tasks.rook.operator.smithi037.stdout:couldn't init storage provider. : exit status 5 2022-03-10T03:33:17.409 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:33:17.411376 I | op-mon: parsing mon endpoints: a=10.100.186.100:6789,b=10.111.132.151:6789,c=10.101.46.100:6789 2022-03-10T03:33:18.103 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:33:18.105107 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T03:33:18.094+0000 7f30e845c380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T03:33:18.103 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10T03:33:18.094+0000 7f30e845c380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T03:33:18.104 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10T03:33:18.094+0000 7f30e845c380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T03:33:18.104 INFO:tasks.rook.operator.smithi037.stdout:couldn't init storage provider. : exit status 5 2022-03-10T03:33:18.191 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:33:18.193325 I | op-mon: parsing mon endpoints: a=10.100.186.100:6789,b=10.111.132.151:6789,c=10.101.46.100:6789 2022-03-10T03:33:18.934 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:33:18.935852 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T03:33:18.926+0000 7fe031b2d380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T03:33:18.934 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10T03:33:18.926+0000 7fe031b2d380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T03:33:18.934 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10T03:33:18.926+0000 7fe031b2d380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T03:33:18.934 INFO:tasks.rook.operator.smithi037.stdout:couldn't init storage provider. : exit status 5 2022-03-10T03:33:19.101 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:33:19.103214 I | op-mon: parsing mon endpoints: a=10.100.186.100:6789,b=10.111.132.151:6789,c=10.101.46.100:6789 2022-03-10T03:33:19.761 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:33:19.763472 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T03:33:19.754+0000 7f8cb9dc6380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T03:33:19.762 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10T03:33:19.754+0000 7f8cb9dc6380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T03:33:19.762 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10T03:33:19.754+0000 7f8cb9dc6380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T03:33:19.762 INFO:tasks.rook.operator.smithi037.stdout:couldn't init storage provider. : exit status 5 2022-03-10T03:33:20.089 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:33:20.091093 I | op-mon: parsing mon endpoints: a=10.100.186.100:6789,b=10.111.132.151:6789,c=10.101.46.100:6789 2022-03-10T03:33:20.778 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:33:20.780072 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T03:33:20.770+0000 7f8129d1a380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T03:33:20.778 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10T03:33:20.770+0000 7f8129d1a380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T03:33:20.779 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10T03:33:20.770+0000 7f8129d1a380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T03:33:20.779 INFO:tasks.rook.operator.smithi037.stdout:couldn't init storage provider. : exit status 5 2022-03-10T03:33:21.425 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:33:21.427359 I | op-mon: parsing mon endpoints: a=10.100.186.100:6789,b=10.111.132.151:6789,c=10.101.46.100:6789 2022-03-10T03:33:22.181 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:33:22.183060 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T03:33:22.174+0000 7f345c36b380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T03:33:22.181 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10T03:33:22.174+0000 7f345c36b380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T03:33:22.181 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10T03:33:22.174+0000 7f345c36b380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T03:33:22.182 INFO:tasks.rook.operator.smithi037.stdout:couldn't init storage provider. : exit status 5 2022-03-10T03:33:23.474 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:33:23.471674 I | op-mon: parsing mon endpoints: a=10.100.186.100:6789,b=10.111.132.151:6789,c=10.101.46.100:6789 2022-03-10T03:33:24.190 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:33:24.191794 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T03:33:24.182+0000 7f795e39d380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T03:33:24.190 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10T03:33:24.182+0000 7f795e39d380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T03:33:24.190 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10T03:33:24.182+0000 7f795e39d380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T03:33:24.191 INFO:tasks.rook.operator.smithi037.stdout:couldn't init storage provider. : exit status 5 2022-03-10T03:33:24.518 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:33:24.520764 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:33:24.531 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:33:24.533998 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:33:26.763 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:33:26.765492 I | op-mon: parsing mon endpoints: a=10.100.186.100:6789,b=10.111.132.151:6789,c=10.101.46.100:6789 2022-03-10T03:33:27.483 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:33:27.485341 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T03:33:27.474+0000 7fa2a343d380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T03:33:27.484 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10T03:33:27.474+0000 7fa2a343d380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T03:33:27.484 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10T03:33:27.474+0000 7fa2a343d380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T03:33:27.484 INFO:tasks.rook.operator.smithi037.stdout:couldn't init storage provider. : exit status 5 2022-03-10T03:33:32.610 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:33:32.611972 I | op-mon: parsing mon endpoints: a=10.100.186.100:6789,b=10.111.132.151:6789,c=10.101.46.100:6789 2022-03-10T03:33:33.359 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:33:33.361340 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T03:33:33.350+0000 7f8347de4380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T03:33:33.359 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10T03:33:33.350+0000 7f8347de4380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T03:33:33.360 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10T03:33:33.350+0000 7f8347de4380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T03:33:33.360 INFO:tasks.rook.operator.smithi037.stdout:couldn't init storage provider. : exit status 5 2022-03-10T03:33:34.651 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:33:34.653522 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:33:34.663 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:33:34.665660 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:33:43.608 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:33:43.610048 I | op-mon: parsing mon endpoints: a=10.100.186.100:6789,b=10.111.132.151:6789,c=10.101.46.100:6789 2022-03-10T03:33:44.401 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:33:44.402938 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T03:33:44.393+0000 7f164069b380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T03:33:44.401 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10T03:33:44.393+0000 7f164069b380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T03:33:44.402 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10T03:33:44.393+0000 7f164069b380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T03:33:44.402 INFO:tasks.rook.operator.smithi037.stdout:couldn't init storage provider. : exit status 5 2022-03-10T03:33:44.764 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:33:44.766560 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:33:44.776 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:33:44.778565 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:33:55.013 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:33:55.015107 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:33:55.024 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:33:55.026509 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:34:04.891 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:34:04.893140 I | op-mon: parsing mon endpoints: a=10.100.186.100:6789,b=10.111.132.151:6789,c=10.101.46.100:6789 2022-03-10T03:34:05.108 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:34:05.110809 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:34:05.118 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:34:05.120673 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:34:05.545 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:34:05.547088 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T03:34:05.537+0000 7ff39367e380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T03:34:05.545 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10T03:34:05.537+0000 7ff39367e380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T03:34:05.546 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10T03:34:05.537+0000 7ff39367e380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T03:34:05.546 INFO:tasks.rook.operator.smithi037.stdout:couldn't init storage provider. : exit status 5 2022-03-10T03:34:15.212 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:34:15.214319 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:34:15.226 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:34:15.228117 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:34:25.330 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:34:25.332190 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:34:25.345 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:34:25.347376 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:34:35.405 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:34:35.407189 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:34:35.418 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:34:35.420656 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:34:45.537 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:34:45.539275 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:34:45.550 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:34:45.551835 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:34:46.512 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:34:46.514020 I | op-mon: parsing mon endpoints: a=10.100.186.100:6789,b=10.111.132.151:6789,c=10.101.46.100:6789 2022-03-10T03:34:47.223 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:34:47.224376 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T03:34:47.212+0000 7fc677db9380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T03:34:47.224 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10T03:34:47.212+0000 7fc677db9380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T03:34:47.224 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10T03:34:47.212+0000 7fc677db9380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T03:34:47.225 INFO:tasks.rook.operator.smithi037.stdout:couldn't init storage provider. : exit status 5 2022-03-10T03:34:55.666 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:34:55.667057 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:34:55.678 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:34:55.679984 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:35:05.751 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:35:05.752692 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:35:05.830 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:35:05.832159 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:35:15.926 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:35:15.927667 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:35:15.943 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:35:15.939863 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:35:26.046 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:35:26.048143 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:35:26.062 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:35:26.063050 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:35:36.137 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:35:36.139006 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:35:36.151 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:35:36.152892 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:35:46.225 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:35:46.226637 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:35:46.238 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:35:46.240165 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:35:56.359 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:35:56.361165 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:35:56.369 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:35:56.371634 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:36:06.493 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:36:06.494522 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:36:06.505 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:36:06.507209 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:36:09.151 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:36:09.152894 I | op-mon: parsing mon endpoints: a=10.100.186.100:6789,b=10.111.132.151:6789,c=10.101.46.100:6789 2022-03-10T03:36:09.824 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:36:09.824852 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T03:36:09.811+0000 7fd84608e380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T03:36:09.825 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10T03:36:09.811+0000 7fd84608e380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T03:36:09.826 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10T03:36:09.811+0000 7fd84608e380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T03:36:09.826 INFO:tasks.rook.operator.smithi037.stdout:couldn't init storage provider. : exit status 5 2022-03-10T03:36:16.628 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:36:16.629453 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:36:16.644 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:36:16.641207 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:36:26.744 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:36:26.745321 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:36:26.757 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:36:26.758901 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:36:36.887 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:36:36.888928 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:36:36.903 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:36:36.903524 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:36:47.046 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:36:47.047749 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:36:47.062 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:36:47.063615 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:36:57.188 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:36:57.189754 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:36:57.198 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:36:57.200062 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:37:07.272 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:37:07.273706 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:37:07.287 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:37:07.287989 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:37:17.381 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:37:17.382455 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:37:17.393 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:37:17.394586 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:37:27.474 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:37:27.475285 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:37:27.491 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:37:27.492479 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:37:37.610 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:37:37.611244 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:37:37.623 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:37:37.624758 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:37:47.688 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:37:47.689813 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:37:47.702 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:37:47.703000 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:37:57.763 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:37:57.764699 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:37:57.804 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:37:57.805801 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:38:07.924 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:38:07.925182 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:38:07.938 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:38:07.939679 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:38:18.052 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:38:18.053617 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:38:18.067 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:38:18.068465 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:38:28.148 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:38:28.144880 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:38:28.159 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:38:28.159862 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:38:38.265 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:38:38.266199 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:38:38.278 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:38:38.279711 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:38:48.400 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:38:48.401213 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:38:48.419 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:38:48.416394 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:38:53.680 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:38:53.680874 I | op-mon: parsing mon endpoints: a=10.100.186.100:6789,b=10.111.132.151:6789,c=10.101.46.100:6789 2022-03-10T03:38:54.385 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:38:54.385873 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T03:38:54.372+0000 7f53e8a90380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T03:38:54.385 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10T03:38:54.372+0000 7f53e8a90380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T03:38:54.386 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10T03:38:54.372+0000 7f53e8a90380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T03:38:54.386 INFO:tasks.rook.operator.smithi037.stdout:couldn't init storage provider. : exit status 5 2022-03-10T03:38:58.495 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:38:58.496033 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:38:58.510 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:38:58.510777 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:39:08.576 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:39:08.572412 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:39:08.587 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:39:08.587978 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:39:18.666 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:39:18.666740 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:39:18.679 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:39:18.680497 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:39:28.798 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:39:28.791665 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:39:28.802 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:39:28.803838 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:39:38.869 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:39:38.869513 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:39:38.881 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:39:38.881920 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:39:48.996 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:39:48.996648 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:39:49.008 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:39:49.009274 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:39:59.122 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:39:59.122864 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:39:59.141 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:39:59.136506 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:40:09.203 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:40:09.204003 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:40:09.213 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:40:09.213903 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:40:19.309 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:40:19.310047 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:40:19.321 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:40:19.321516 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:40:29.403 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:40:29.404358 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:40:29.415 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:40:29.416715 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:40:39.734 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:40:39.735105 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:40:39.746 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:40:39.747014 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:40:49.843 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:40:49.843427 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:40:49.853 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:40:49.854362 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:40:59.919 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:40:59.919989 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:40:59.932 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:40:59.933631 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:41:10.010 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:41:10.011190 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:41:10.021 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:41:10.022491 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:41:20.155 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:41:20.155617 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:41:20.168 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:41:20.168562 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:41:30.239 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:41:30.239899 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:41:30.250 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:41:30.251593 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:41:40.320 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:41:40.321218 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:41:40.333 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:41:40.333955 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:41:50.450 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:41:50.450518 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:41:50.464 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:41:50.465177 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:42:00.577 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:42:00.578420 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:42:00.590 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:42:00.591105 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:42:10.688 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:42:10.688561 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:42:10.701 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:42:10.701867 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:42:20.794 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:42:20.795107 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:42:20.808 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:42:20.808748 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:42:30.930 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:42:30.930615 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:42:30.945 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:42:30.945214 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:42:41.045 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:42:41.045971 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:42:41.060 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:42:41.060830 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:42:51.127 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:42:51.127340 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:42:51.137 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:42:51.137870 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:43:01.210 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:43:01.210763 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:43:01.224 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:43:01.224394 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:43:11.335 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:43:11.335143 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:43:11.346 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:43:11.347081 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:43:21.450 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:43:21.450687 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:43:21.463 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:43:21.463982 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:43:31.549 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:43:31.549600 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:43:31.561 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:43:31.562349 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:43:41.717 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:43:41.718126 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:43:41.727 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:43:41.728156 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:43:51.848 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:43:51.849174 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:43:51.867 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:43:51.867857 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:44:01.959 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:44:01.959347 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:44:01.971 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:44:01.972030 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:44:12.037 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:44:12.037620 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:44:12.050 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:44:12.051056 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:44:22.110 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:44:22.111076 I | op-mon: parsing mon endpoints: a=10.100.186.100:6789,b=10.111.132.151:6789,c=10.101.46.100:6789 2022-03-10T03:44:22.162 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:44:22.162607 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:44:22.173 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:44:22.174097 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:44:22.799 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:44:22.798583 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T03:44:22.790+0000 7fc51f87d380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T03:44:22.800 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10T03:44:22.790+0000 7fc51f87d380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T03:44:22.800 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10T03:44:22.790+0000 7fc51f87d380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T03:44:22.800 INFO:tasks.rook.operator.smithi037.stdout:couldn't init storage provider. : exit status 5 2022-03-10T03:44:32.235 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:44:32.235150 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:44:32.247 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:44:32.248131 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:44:42.354 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:44:42.354180 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:44:42.368 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:44:42.369317 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:44:52.476 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:44:52.476597 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:44:52.489 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:44:52.489980 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:45:02.609 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:45:02.609041 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:45:02.623 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:45:02.624195 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:45:12.702 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:45:12.702306 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:45:12.716 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:45:12.716600 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:45:22.823 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:45:22.823852 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:45:22.837 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:45:22.837627 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:45:32.909 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:45:32.909916 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:45:32.924 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:45:32.924562 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:45:42.992 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:45:42.992585 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:45:43.003 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:45:43.003933 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:45:53.084 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:45:53.083876 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:45:53.097 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:45:53.097392 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:46:03.199 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:46:03.199484 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:46:03.211 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:46:03.212133 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:46:13.316 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:46:13.316455 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:46:13.333 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:46:13.333456 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:46:23.441 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:46:23.440923 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:46:23.451 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:46:23.451825 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:46:33.515 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:46:33.515030 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:46:33.528 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:46:33.528122 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:46:43.614 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:46:43.613966 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:46:43.629 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:46:43.629040 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:46:53.718 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:46:53.717983 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:46:53.732 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:46:53.732803 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:47:03.910 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:47:03.910180 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:47:03.922 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:47:03.922838 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:47:14.031 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:47:14.030828 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:47:14.044 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:47:14.043728 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:47:24.186 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:47:24.186367 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:47:24.198 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:47:24.198438 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:47:34.264 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:47:34.263595 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:47:34.277 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:47:34.276829 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:47:44.378 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:47:44.378195 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:47:44.396 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:47:44.396106 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:47:54.466 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:47:54.465772 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:47:54.478 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:47:54.478733 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:48:04.576 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:48:04.575962 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:48:04.589 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:48:04.588828 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:48:14.684 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:48:14.683534 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:48:14.698 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:48:14.698217 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:48:24.765 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:48:24.764603 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:48:24.783 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:48:24.782780 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:48:34.851 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:48:34.850860 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:48:34.865 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:48:34.864809 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:48:44.984 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:48:44.984036 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:48:44.999 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:48:44.999102 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:48:55.122 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:48:55.122431 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:48:55.137 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:48:55.137497 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:49:05.267 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:49:05.264318 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:49:05.277 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:49:05.276747 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:49:15.399 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:49:15.397382 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:49:15.410 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:49:15.409657 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:49:25.500 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:49:25.500112 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:49:25.513 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:49:25.513270 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:49:35.617 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:49:35.615253 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:49:35.633 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:49:35.631954 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:49:45.707 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:49:45.706991 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:49:45.722 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:49:45.721753 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:49:55.793 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:49:55.793282 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:49:55.807 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:49:55.807508 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:50:05.871 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:50:05.870508 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:50:05.882 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:50:05.882523 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:50:15.992 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:50:15.991360 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:50:16.007 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:50:16.007653 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:50:26.088 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:50:26.087893 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:50:26.101 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:50:26.101357 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:50:36.175 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:50:36.174228 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:50:36.188 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:50:36.188283 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:50:46.268 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:50:46.267435 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:50:46.284 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:50:46.283390 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:50:56.355 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:50:56.354663 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:50:56.404 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:50:56.403627 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:51:06.522 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:51:06.521440 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:51:06.535 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:51:06.534612 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:51:16.642 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:51:16.641916 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:51:16.656 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:51:16.656204 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:51:26.771 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:51:26.770144 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:51:26.784 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:51:26.783461 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:51:36.879 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:51:36.878559 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:51:36.896 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:51:36.896356 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:51:47.029 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:51:47.028289 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:51:47.043 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:51:47.042782 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:51:57.126 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:51:57.125288 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:51:57.137 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:51:57.136932 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:52:07.215 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:52:07.214289 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:52:07.227 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:52:07.226746 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:52:17.331 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:52:17.331014 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:52:17.344 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:52:17.343835 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:52:27.422 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:52:27.421450 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:52:27.437 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:52:27.437003 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:52:37.544 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:52:37.543509 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:52:37.558 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:52:37.557795 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:52:47.678 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:52:47.676065 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:52:47.691 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:52:47.690676 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:52:57.768 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:52:57.767233 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:52:57.781 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:52:57.780816 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:53:07.845 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:53:07.842257 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:53:07.855 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:53:07.854550 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:53:17.960 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:53:17.960015 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:53:17.971 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:53:17.970662 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:53:28.065 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:53:28.064469 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:53:28.120 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:53:28.119357 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:53:38.185 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:53:38.184220 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:53:38.196 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:53:38.195161 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:53:48.266 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:53:48.265214 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:53:48.280 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:53:48.279565 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:53:58.378 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:53:58.377445 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:53:58.400 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:53:58.399297 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:54:08.500 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:54:08.499880 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:54:08.514 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:54:08.514233 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:54:18.631 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:54:18.629177 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:54:18.647 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:54:18.642458 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:54:28.768 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:54:28.767964 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:54:28.782 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:54:28.782280 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:54:38.878 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:54:38.877261 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:54:38.893 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:54:38.892415 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:54:48.958 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:54:48.957810 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:54:48.971 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:54:48.970450 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:54:59.047 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:54:59.042614 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:54:59.094 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:54:59.093350 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:55:09.159 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:55:09.157849 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:55:09.172 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:55:09.171319 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:55:18.166 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:55:18.165454 I | op-mon: parsing mon endpoints: a=10.100.186.100:6789,b=10.111.132.151:6789,c=10.101.46.100:6789 2022-03-10T03:55:18.949 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:55:18.948559 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T03:55:18.938+0000 7f92a1461380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T03:55:18.950 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10T03:55:18.938+0000 7f92a1461380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T03:55:18.950 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10T03:55:18.938+0000 7f92a1461380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T03:55:18.951 INFO:tasks.rook.operator.smithi037.stdout:couldn't init storage provider. : exit status 5 2022-03-10T03:55:19.244 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:55:19.243435 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:55:19.257 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:55:19.257198 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:55:29.329 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:55:29.328663 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:55:29.343 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:55:29.342449 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:55:39.414 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:55:39.413709 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:55:39.427 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:55:39.426967 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:55:49.503 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:55:49.502337 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:55:49.517 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:55:49.516494 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:55:59.599 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:55:59.598088 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:55:59.611 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:55:59.610267 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:56:09.706 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:56:09.705122 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:56:09.717 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:56:09.716917 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:56:19.791 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:56:19.790662 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:56:19.844 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:56:19.843671 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:56:29.962 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:56:29.961023 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:56:29.974 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:56:29.973292 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:56:40.155 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:56:40.154837 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:56:40.173 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:56:40.173091 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:56:50.287 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:56:50.286584 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:56:50.300 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:56:50.299993 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:57:00.383 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:57:00.383009 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:57:00.396 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:57:00.395485 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:57:10.469 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:57:10.468635 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:57:10.481 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:57:10.481077 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:57:20.551 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:57:20.550431 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:57:20.563 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:57:20.562622 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:57:30.668 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:57:30.666962 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:57:30.681 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:57:30.680735 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:57:40.755 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:57:40.753878 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:57:40.821 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:57:40.820118 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:57:50.931 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:57:50.930070 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:57:50.944 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:57:50.943927 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:58:01.057 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:58:01.056849 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:58:01.069 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:58:01.068338 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:58:11.168 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:58:11.167133 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:58:11.180 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:58:11.180195 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:58:21.275 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:58:21.274022 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:58:21.288 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:58:21.287308 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:58:31.345 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:58:31.341444 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:58:31.359 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:58:31.358710 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:58:41.451 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:58:41.450009 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:58:41.465 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:58:41.464784 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:58:51.585 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:58:51.584029 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:58:51.600 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:58:51.599761 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:59:01.700 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:59:01.699087 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:59:01.716 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:59:01.715365 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:59:11.816 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:59:11.815071 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:59:11.827 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:59:11.826803 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:59:21.969 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:59:21.968176 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:59:21.978 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:59:21.978119 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:59:32.047 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:59:32.045939 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:59:32.057 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:59:32.057183 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:59:42.168 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:59:42.167147 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:59:42.182 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:59:42.181680 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:59:52.423 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:59:52.422707 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:59:52.438 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 03:59:52.437278 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:00:02.504 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:00:02.503420 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:00:02.518 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:00:02.517600 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:00:12.630 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:00:12.629357 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:00:12.643 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:00:12.642396 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:00:22.779 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:00:22.778276 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:00:22.791 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:00:22.790838 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:00:32.859 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:00:32.858123 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:00:32.871 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:00:32.870472 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:00:42.961 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:00:42.960483 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:00:43.006 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:00:43.005590 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:00:53.086 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:00:53.085743 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:00:53.131 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:00:53.130558 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:01:03.249 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:01:03.248241 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:01:03.265 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:01:03.264291 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:01:13.359 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:01:13.358123 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:01:13.371 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:01:13.371111 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:01:23.473 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:01:23.472347 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:01:23.485 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:01:23.484901 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:01:33.563 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:01:33.562220 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:01:33.618 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:01:33.617903 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:01:43.747 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:01:43.746155 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:01:43.762 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:01:43.761530 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:01:53.840 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:01:53.839127 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:01:53.907 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:01:53.905400 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:02:04.014 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:02:04.012222 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:02:04.026 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:02:04.025964 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:02:14.233 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:02:14.232264 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:02:14.247 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:02:14.246384 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:02:24.314 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:02:24.313227 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:02:24.329 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:02:24.328045 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:02:34.426 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:02:34.425204 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:02:34.439 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:02:34.438621 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:02:44.536 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:02:44.534701 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:02:44.550 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:02:44.549095 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:02:54.682 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:02:54.681134 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:02:54.694 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:02:54.693661 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:03:04.791 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:03:04.790777 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:03:04.809 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:03:04.808432 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:03:14.870 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:03:14.869084 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:03:14.911 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:03:14.910785 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:03:25.005 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:03:25.003722 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:03:25.016 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:03:25.014948 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:03:35.083 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:03:35.082496 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:03:35.094 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:03:35.093725 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:03:45.164 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:03:45.162994 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:03:45.177 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:03:45.175736 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:03:55.329 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:03:55.327682 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:03:55.341 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:03:55.340396 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:04:05.482 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:04:05.480876 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:04:05.495 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:04:05.494510 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:04:15.584 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:04:15.582641 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:04:15.596 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:04:15.595002 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:04:25.735 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:04:25.733773 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:04:25.750 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:04:25.749401 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:04:35.823 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:04:35.822231 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:04:35.836 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:04:35.834975 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:04:45.900 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:04:45.899409 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:04:45.912 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:04:45.910883 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:04:55.976 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:04:55.975385 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:04:55.986 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:04:55.985356 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:05:06.108 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:05:06.106537 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:05:06.120 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:05:06.119371 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:05:16.240 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:05:16.238971 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:05:16.254 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:05:16.252774 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:05:26.367 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:05:26.366011 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:05:26.386 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:05:26.379981 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:05:36.444 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:05:36.440129 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:05:36.456 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:05:36.454958 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:05:46.568 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:05:46.567342 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:05:46.583 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:05:46.581827 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:05:56.689 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:05:56.688215 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:05:56.704 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:05:56.703106 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:06:06.805 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:06:06.802157 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:06:06.816 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:06:06.814523 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:06:16.920 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:06:16.918595 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:06:16.933 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:06:16.931858 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:06:27.030 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:06:27.028491 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:06:27.042 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:06:27.041011 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:06:37.173 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:06:37.171805 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:06:37.185 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:06:37.184634 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:06:47.284 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:06:47.282552 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:06:47.299 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:06:47.297581 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:06:57.420 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:06:57.418370 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:06:57.432 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:06:57.431049 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:07:07.500 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:07:07.498693 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:07:07.513 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:07:07.512132 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:07:17.583 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:07:17.581495 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:07:17.597 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:07:17.595666 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:07:27.694 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:07:27.692638 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:07:27.708 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:07:27.707426 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:07:37.829 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:07:37.827846 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:07:37.844 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:07:37.841557 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:07:47.912 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:07:47.910602 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:07:47.924 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:07:47.922969 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:07:58.042 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:07:58.041130 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:07:58.053 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:07:58.052170 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:08:08.170 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:08:08.169040 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:08:08.184 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:08:08.182795 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:08:18.290 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:08:18.289234 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:08:18.303 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:08:18.302577 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:08:28.455 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:08:28.453730 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:08:28.467 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:08:28.466210 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:08:38.580 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:08:38.579162 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:08:38.591 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:08:38.590134 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:08:48.700 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:08:48.699180 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:08:48.716 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:08:48.715255 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:08:58.844 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:08:58.842905 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:08:58.855 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:08:58.854562 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:09:08.990 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:09:08.989117 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:09:09.004 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:09:09.003308 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:09:19.106 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:09:19.104728 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:09:19.116 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:09:19.115672 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:09:29.240 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:09:29.238783 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:09:29.253 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:09:29.251681 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:09:39.322 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:09:39.321061 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:09:39.336 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:09:39.335468 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:09:49.441 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:09:49.440166 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:09:49.456 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:09:49.455152 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:09:59.568 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:09:59.567124 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:09:59.581 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:09:59.580801 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:10:09.652 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:10:09.651314 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:10:09.667 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:10:09.666617 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:10:19.729 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:10:19.728136 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:10:19.773 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:10:19.772105 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:10:29.909 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:10:29.907904 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:10:29.922 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:10:29.921355 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:10:40.007 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:10:40.006677 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:10:40.024 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:10:40.023111 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:10:50.118 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:10:50.116895 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:10:50.129 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:10:50.128753 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:11:00.267 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:11:00.266585 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:11:00.283 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:11:00.283058 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:11:10.387 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:11:10.386283 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:11:10.400 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:11:10.399647 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:11:20.470 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:11:20.469019 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:11:20.543 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:11:20.542580 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:11:30.640 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:11:30.639380 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:11:30.653 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:11:30.652415 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:11:40.730 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:11:40.729432 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:11:40.742 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:11:40.741559 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:11:50.812 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:11:50.811322 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:11:50.825 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:11:50.824944 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:11:58.958 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:11:58.957361 I | op-mon: parsing mon endpoints: a=10.100.186.100:6789,b=10.111.132.151:6789,c=10.101.46.100:6789 2022-03-10T04:11:59.641 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:11:59.640004 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T04:11:59.630+0000 7f6ae018c380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T04:11:59.642 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10T04:11:59.630+0000 7f6ae018c380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T04:11:59.642 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10T04:11:59.630+0000 7f6ae018c380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T04:11:59.642 INFO:tasks.rook.operator.smithi037.stdout:couldn't init storage provider. : exit status 5 2022-03-10T04:12:00.894 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:12:00.893384 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:12:00.906 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:12:00.905813 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:12:10.973 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:12:10.971506 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:12:10.986 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:12:10.985837 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:12:21.098 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:12:21.097181 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:12:21.109 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:12:21.109006 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:12:31.215 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:12:31.214183 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:12:31.228 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:12:31.227511 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:12:41.335 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:12:41.334217 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:12:41.361 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:12:41.360402 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:12:51.440 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:12:51.439310 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:12:51.508 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:12:51.507835 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:13:01.579 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:13:01.578164 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:13:01.591 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:13:01.591025 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:13:11.728 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:13:11.726245 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:13:11.740 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:13:11.739550 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:13:21.834 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:13:21.833310 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:13:21.847 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:13:21.846652 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:13:31.964 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:13:31.964079 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:13:31.988 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:13:31.987331 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:13:42.254 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:13:42.253535 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:13:42.268 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:13:42.267423 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:13:52.356 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:13:52.355789 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:13:52.371 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:13:52.370540 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:14:02.526 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:14:02.525175 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:14:02.540 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:14:02.539957 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:14:12.756 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:14:12.755699 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:14:12.771 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:14:12.770477 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:14:22.854 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:14:22.853260 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:14:22.865 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:14:22.864849 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:14:32.930 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:14:32.929654 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:14:32.942 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:14:32.942162 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:14:43.011 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:14:43.011134 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:14:43.025 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:14:43.024306 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:14:53.170 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:14:53.170002 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:14:53.195 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:14:53.195238 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:15:03.272 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:15:03.271300 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:15:03.287 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:15:03.286477 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:15:13.396 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:15:13.395963 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:15:13.410 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:15:13.409496 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:15:23.500 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:15:23.499594 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:15:23.512 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:15:23.511699 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:15:33.576 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:15:33.576072 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:15:33.587 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:15:33.587124 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:15:43.732 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:15:43.730814 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:15:43.745 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:15:43.744545 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:15:53.808 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:15:53.807600 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:15:53.826 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:15:53.826024 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:16:03.894 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:16:03.894007 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:16:03.907 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:16:03.906645 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:16:13.980 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:16:13.979852 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:16:13.992 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:16:13.991903 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:16:24.086 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:16:24.085757 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:16:24.098 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:16:24.097624 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:16:34.199 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:16:34.198182 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:16:34.209 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:16:34.209094 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:16:44.317 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:16:44.316545 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:16:44.331 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:16:44.330616 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:16:54.441 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:16:54.440598 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:16:54.454 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:16:54.453924 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:17:04.524 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:17:04.523307 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:17:04.590 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:17:04.589534 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:17:14.698 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:17:14.697050 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:17:14.713 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:17:14.712701 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:17:24.821 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:17:24.820285 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:17:24.835 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:17:24.834060 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:17:34.911 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:17:34.911156 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:17:34.923 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:17:34.922783 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:17:45.022 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:17:45.021949 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:17:45.034 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:17:45.034327 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:17:55.095 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:17:55.094970 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:17:55.108 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:17:55.108431 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:18:05.215 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:18:05.213779 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:18:05.229 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:18:05.228240 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:18:15.359 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:18:15.359062 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:18:15.374 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:18:15.373448 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:18:25.448 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:18:25.447750 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:18:25.463 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:18:25.463244 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:18:35.541 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:18:35.540938 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:18:35.579 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:18:35.578777 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:18:45.655 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:18:45.654432 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:18:45.671 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:18:45.671202 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:18:55.784 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:18:55.783690 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:18:55.801 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:18:55.797206 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:19:05.886 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:19:05.885703 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:19:05.898 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:19:05.898065 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:19:15.985 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:19:15.984384 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:19:15.997 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:19:15.997378 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:19:26.069 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:19:26.063454 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:19:26.120 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:19:26.119699 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:19:36.202 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:19:36.201391 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:19:36.215 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:19:36.214800 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:19:46.331 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:19:46.331121 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:19:46.346 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:19:46.346164 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:19:56.457 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:19:56.456283 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:19:56.470 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:19:56.470239 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:20:06.561 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:20:06.561210 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:20:06.613 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:20:06.612974 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:20:16.702 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:20:16.701804 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:20:16.716 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:20:16.716282 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:20:26.787 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:20:26.786499 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:20:26.797 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:20:26.797410 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:20:36.862 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:20:36.861064 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:20:36.873 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:20:36.872536 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:20:47.035 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:20:47.034119 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:20:47.049 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:20:47.048599 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:20:57.140 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:20:57.140095 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:20:57.342 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:20:57.342215 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:21:07.440 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:21:07.439587 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:21:07.452 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:21:07.452451 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:21:17.547 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:21:17.546154 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:21:17.560 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:21:17.559449 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:21:27.676 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:21:27.676111 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:21:27.690 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:21:27.689537 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:21:37.783 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:21:37.782843 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:21:37.797 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:21:37.797405 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:21:47.922 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:21:47.921681 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:21:47.933 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:21:47.933249 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:21:58.055 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:21:58.054244 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:21:58.066 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:21:58.066089 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:22:08.186 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:22:08.186139 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:22:08.202 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:22:08.201727 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:22:18.255 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:22:18.255071 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:22:18.268 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:22:18.268175 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:22:28.374 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:22:28.373685 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:22:28.386 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:22:28.386242 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:22:38.507 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:22:38.506254 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:22:38.518 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:22:38.518015 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:22:48.619 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:22:48.619344 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:22:48.634 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:22:48.633165 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:22:58.700 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:22:58.699517 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:22:58.743 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:22:58.742327 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:23:08.846 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:23:08.845908 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:23:08.860 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:23:08.860003 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:23:18.969 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:23:18.968674 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:23:18.983 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:23:18.983017 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:23:29.057 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:23:29.056403 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:23:29.070 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:23:29.069684 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:23:39.174 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:23:39.172380 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:23:39.183 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:23:39.182833 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:23:49.257 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:23:49.256683 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:23:49.306 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:23:49.305821 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:23:59.376 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:23:59.375902 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:23:59.389 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:23:59.389367 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:24:09.499 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:24:09.498599 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:24:09.517 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:24:09.516837 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:24:19.630 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:24:19.629268 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:24:19.644 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:24:19.643537 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:24:29.713 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:24:29.712348 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:24:29.726 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:24:29.725958 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:24:39.799 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:24:39.798340 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:24:39.813 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:24:39.813078 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:24:49.880 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:24:49.879688 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:24:49.893 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:24:49.892842 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:24:59.968 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:24:59.967442 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:24:59.983 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:24:59.982867 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:25:10.083 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:25:10.082478 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:25:10.094 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:25:10.093513 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:25:20.211 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:25:20.210002 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:25:20.224 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:25:20.223800 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:25:30.318 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:25:30.317981 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:25:30.369 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:25:30.368641 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:25:40.488 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:25:40.487425 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:25:40.515 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:25:40.515091 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:25:50.612 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:25:50.612153 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:25:50.627 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:25:50.626752 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:26:00.741 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:26:00.740263 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:26:00.754 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:26:00.753683 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:26:10.859 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:26:10.858728 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:26:10.880 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:26:10.879906 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:26:21.017 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:26:21.016780 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:26:21.032 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:26:21.032164 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:26:31.144 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:26:31.143993 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:26:31.160 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:26:31.159630 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:26:41.295 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:26:41.294483 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:26:41.306 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:26:41.305863 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:26:51.397 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:26:51.396487 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:26:51.411 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:26:51.410663 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:27:01.510 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:27:01.510222 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:27:01.525 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:27:01.524986 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:27:11.628 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:27:11.628033 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:27:11.641 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:27:11.640693 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:27:21.720 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:27:21.719210 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:27:21.735 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:27:21.734679 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:27:31.802 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:27:31.801593 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:27:31.813 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:27:31.811950 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:27:41.964 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:27:41.962600 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:27:41.976 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:27:41.975025 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:27:52.105 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:27:52.104816 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:27:52.116 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:27:52.115734 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:28:02.208 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:28:02.208051 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:28:02.222 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:28:02.221749 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:28:12.309 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:28:12.308599 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:28:12.323 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:28:12.322641 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:28:22.418 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:28:22.418096 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:28:22.432 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:28:22.431864 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:28:32.561 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:28:32.560553 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:28:32.575 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:28:32.575440 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:28:39.653 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:28:39.652194 I | op-mon: parsing mon endpoints: a=10.100.186.100:6789,b=10.111.132.151:6789,c=10.101.46.100:6789 2022-03-10T04:28:40.313 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:28:40.312669 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T04:28:40.299+0000 7f4aed5b0380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T04:28:40.314 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10T04:28:40.299+0000 7f4aed5b0380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T04:28:40.314 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10T04:28:40.299+0000 7f4aed5b0380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T04:28:40.314 INFO:tasks.rook.operator.smithi037.stdout:couldn't init storage provider. : exit status 5 2022-03-10T04:28:42.668 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:28:42.668206 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:28:42.679 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:28:42.679512 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:28:52.748 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:28:52.747648 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:28:52.782 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:28:52.782132 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:29:02.913 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:29:02.913032 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:29:02.925 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:29:02.924804 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:29:13.032 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:29:13.031408 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:29:13.048 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:29:13.048117 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:29:23.121 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:29:23.120443 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:29:23.132 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:29:23.132126 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:29:33.238 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:29:33.237761 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:29:33.252 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:29:33.252351 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:29:43.354 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:29:43.353907 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:29:43.364 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:29:43.364582 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:29:53.436 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:29:53.435347 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:29:53.447 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:29:53.446691 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:30:03.506 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:30:03.505489 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:30:03.518 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:30:03.518010 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:30:13.576 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:30:13.572660 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:30:13.582 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:30:13.581966 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:30:23.677 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:30:23.677275 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:30:23.690 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:30:23.689990 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:30:33.807 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:30:33.806539 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:30:33.821 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:30:33.821549 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:30:43.952 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:30:43.951591 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:30:43.966 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:30:43.966284 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:30:54.071 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:30:54.070607 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:30:54.084 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:30:54.083745 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:31:04.174 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:31:04.173663 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:31:04.188 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:31:04.187753 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:31:14.294 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:31:14.293068 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:31:14.307 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:31:14.307444 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:31:24.430 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:31:24.429293 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:31:24.443 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:31:24.443079 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:31:34.513 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:31:34.512534 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:31:34.524 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:31:34.523850 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:31:44.627 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:31:44.627249 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:31:44.641 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:31:44.641234 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:31:54.709 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:31:54.708617 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:31:54.722 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:31:54.722236 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:32:04.791 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:32:04.790738 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:32:04.803 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:32:04.803012 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:32:14.897 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:32:14.896573 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:32:14.909 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:32:14.907873 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:32:25.003 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:32:25.002397 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:32:25.019 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:32:25.018604 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:32:35.125 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:32:35.125080 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:32:35.138 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:32:35.137742 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:32:45.221 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:32:45.220142 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:32:45.235 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:32:45.234487 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:32:55.348 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:32:55.347405 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:32:55.363 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:32:55.362445 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:33:05.476 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:33:05.475680 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:33:05.495 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:33:05.494449 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:33:15.572 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:33:15.571545 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:33:15.585 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:33:15.584583 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:33:25.691 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:33:25.690654 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:33:25.710 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:33:25.708421 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:33:35.807 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:33:35.806015 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:33:35.817 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:33:35.817098 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:33:45.906 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:33:45.905650 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:33:45.922 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:33:45.921851 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:33:55.988 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:33:55.988090 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:33:56.058 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:33:56.057643 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:34:06.216 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:34:06.215312 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:34:06.227 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:34:06.226855 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:34:16.364 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:34:16.363815 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:34:16.401 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:34:16.400589 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:34:26.488 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:34:26.487847 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:34:26.500 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:34:26.499465 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:34:36.573 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:34:36.572508 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:34:36.585 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:34:36.584663 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:34:46.709 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:34:46.708189 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:34:46.723 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:34:46.722270 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:34:56.827 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:34:56.826002 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:34:56.838 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:34:56.838037 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:35:06.906 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:35:06.905131 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:35:06.921 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:35:06.920304 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:35:17.039 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:35:17.035813 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:35:17.049 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:35:17.048821 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:35:27.165 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:35:27.163663 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:35:27.177 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:35:27.176890 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:35:37.302 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:35:37.301325 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:35:37.312 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:35:37.311821 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:35:47.391 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:35:47.391201 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:35:47.402 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:35:47.401788 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:35:57.470 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:35:57.469463 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:35:57.490 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:35:57.489468 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:36:07.622 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:36:07.621392 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:36:07.635 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:36:07.635211 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:36:17.707 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:36:17.704631 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:36:17.718 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:36:17.718068 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:36:27.833 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:36:27.832773 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:36:27.848 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:36:27.847817 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:36:37.946 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:36:37.945719 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:36:37.960 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:36:37.959937 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:36:48.055 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:36:48.054387 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:36:48.098 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:36:48.098075 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:36:58.184 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:36:58.183993 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:36:58.203 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:36:58.203107 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:37:08.289 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:37:08.288208 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:37:08.305 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:37:08.303010 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:37:18.502 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:37:18.501361 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:37:18.515 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:37:18.513990 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:37:28.643 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:37:28.642805 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:37:28.655 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:37:28.654836 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:37:38.728 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:37:38.727929 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:37:38.784 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:37:38.783932 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:37:48.896 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:37:48.895358 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:37:48.910 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:37:48.909937 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:37:59.021 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:37:59.020479 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:37:59.036 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:37:59.034134 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:38:09.113 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:38:09.112393 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:38:09.156 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:38:09.155817 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:38:19.267 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:38:19.266607 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:38:19.281 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:38:19.280387 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:38:29.397 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:38:29.396276 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:38:29.413 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:38:29.412360 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:38:39.496 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:38:39.494940 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:38:39.507 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:38:39.506690 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:38:49.596 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:38:49.595696 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:38:49.612 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:38:49.611327 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:38:59.692 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:38:59.691327 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:38:59.705 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:38:59.703330 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:39:09.829 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:39:09.828975 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:39:09.842 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:39:09.842163 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:39:19.918 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:39:19.917307 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:39:19.962 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:39:19.962165 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:39:30.030 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:39:30.029507 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:39:30.042 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:39:30.041718 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:39:40.154 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:39:40.153373 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:39:40.167 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:39:40.166624 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:39:50.259 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:39:50.258493 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:39:50.272 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:39:50.271359 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:40:00.343 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:40:00.341944 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:40:00.355 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:40:00.354519 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:40:10.546 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:40:10.545816 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:40:10.559 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:40:10.559047 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:40:20.668 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:40:20.667259 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:40:20.681 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:40:20.680754 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:40:30.804 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:40:30.802906 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:40:30.817 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:40:30.815853 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:40:40.921 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:40:40.919367 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:40:40.940 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:40:40.939021 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:40:51.013 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:40:51.012870 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:40:51.026 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:40:51.025540 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:41:01.173 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:41:01.172320 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:41:01.186 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:41:01.185501 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:41:11.254 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:41:11.251802 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:41:11.267 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:41:11.267503 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:41:21.332 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:41:21.329462 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:41:21.342 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:41:21.341296 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:41:31.473 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:41:31.472150 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:41:31.484 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:41:31.484195 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:41:41.596 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:41:41.595030 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:41:41.611 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:41:41.610357 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:41:51.717 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:41:51.716712 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:41:51.732 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:41:51.730526 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:42:01.834 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:42:01.833872 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:42:01.846 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:42:01.845920 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:42:11.961 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:42:11.960696 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:42:11.977 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:42:11.976708 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:42:22.100 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:42:22.099817 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:42:22.113 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:42:22.113245 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:42:32.178 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:42:32.177808 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:42:32.191 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:42:32.190542 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:42:42.318 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:42:42.317251 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:42:42.329 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:42:42.328670 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:42:52.441 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:42:52.437430 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:42:52.451 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:42:52.450552 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:43:02.547 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:43:02.546555 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:43:02.559 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:43:02.559360 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:43:12.634 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:43:12.634293 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:43:12.648 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:43:12.647889 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:43:22.765 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:43:22.764679 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:43:22.778 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:43:22.777916 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:43:32.852 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:43:32.851983 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:43:32.899 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:43:32.898337 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:43:42.997 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:43:42.996349 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:43:43.010 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:43:43.009937 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:43:53.123 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:43:53.121753 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:43:53.136 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:43:53.135874 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:44:03.253 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:44:03.252593 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:44:03.267 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:44:03.266232 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:44:13.346 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:44:13.346176 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:44:13.358 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:44:13.358398 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:44:23.459 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:44:23.458968 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:44:23.469 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:44:23.469337 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:44:33.569 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:44:33.568204 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:44:33.580 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:44:33.580019 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:44:43.655 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:44:43.654358 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:44:43.703 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:44:43.703045 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:44:53.777 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:44:53.776416 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:44:53.791 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:44:53.790395 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:45:03.907 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:45:03.907050 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:45:03.917 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:45:03.917058 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:45:14.049 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:45:14.048416 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:45:14.066 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:45:14.065730 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:45:20.326 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:45:20.325390 I | op-mon: parsing mon endpoints: a=10.100.186.100:6789,b=10.111.132.151:6789,c=10.101.46.100:6789 2022-03-10T04:45:20.989 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:45:20.988512 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T04:45:20.975+0000 7f8e76150380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T04:45:20.990 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10T04:45:20.975+0000 7f8e76150380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T04:45:20.990 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10T04:45:20.975+0000 7f8e76150380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T04:45:20.990 INFO:tasks.rook.operator.smithi037.stdout:couldn't init storage provider. : exit status 5 2022-03-10T04:45:24.195 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:45:24.195362 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:45:24.210 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:45:24.210004 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:45:34.321 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:45:34.321079 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:45:34.333 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:45:34.333293 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:45:44.446 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:45:44.446303 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:45:44.459 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:45:44.459483 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:45:54.569 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:45:54.568288 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:45:54.583 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:45:54.582685 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:46:04.705 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:46:04.704722 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:46:04.720 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:46:04.720022 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:46:14.839 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:46:14.838711 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:46:14.852 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:46:14.852267 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:46:24.968 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:46:24.968366 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:46:24.979 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:46:24.979248 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:46:35.135 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:46:35.135009 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:46:35.147 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:46:35.146973 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:46:45.274 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:46:45.273689 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:46:45.287 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:46:45.286724 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:46:55.406 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:46:55.405485 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:46:55.419 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:46:55.419127 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:47:05.525 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:47:05.524252 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:47:05.535 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:47:05.534715 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:47:15.650 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:47:15.649646 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:47:15.664 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:47:15.662444 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:47:25.758 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:47:25.757829 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:47:25.769 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:47:25.769374 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:47:35.911 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:47:35.911338 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:47:35.923 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:47:35.922596 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:47:46.040 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:47:46.039972 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:47:46.052 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:47:46.052940 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:47:56.118 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:47:56.118264 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:47:56.183 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:47:56.183348 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:48:06.252 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:48:06.251517 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:48:06.264 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:48:06.264798 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:48:16.377 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:48:16.376753 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:48:16.390 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:48:16.390563 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:48:26.522 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:48:26.521974 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:48:26.535 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:48:26.535184 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:48:36.633 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:48:36.632687 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:48:36.645 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:48:36.645420 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:48:46.713 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:48:46.702307 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:48:46.716 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:48:46.715977 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:48:56.837 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:48:56.836775 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:48:56.849 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:48:56.849131 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:49:06.923 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:49:06.922634 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:49:06.935 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:49:06.934835 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:49:17.009 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:49:17.008703 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:49:17.023 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:49:17.023767 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:49:27.151 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:49:27.151203 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:49:27.167 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:49:27.166282 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:49:37.264 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:49:37.264412 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:49:37.277 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:49:37.277252 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:49:47.363 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:49:47.362764 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:49:47.376 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:49:47.375756 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:49:57.457 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:49:57.455645 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:49:57.472 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:49:57.472373 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:50:07.537 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:50:07.537168 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:50:07.583 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:50:07.583236 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:50:17.686 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:50:17.686154 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:50:17.702 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:50:17.702625 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:50:27.835 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:50:27.834523 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:50:27.850 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:50:27.850018 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:50:37.944 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:50:37.944501 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:50:37.961 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:50:37.960767 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:50:48.110 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:50:48.110306 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:50:48.123 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:50:48.123160 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:50:58.248 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:50:58.248060 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:50:58.261 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:50:58.261931 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:51:08.332 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:51:08.331691 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:51:08.350 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:51:08.350547 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:51:18.460 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:51:18.460123 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:51:18.473 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:51:18.473668 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:51:28.538 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:51:28.538021 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:51:28.552 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:51:28.552705 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:51:38.687 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:51:38.687093 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:51:38.700 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:51:38.699939 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:51:48.766 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:51:48.766005 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:51:48.828 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:51:48.828084 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:51:58.899 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:51:58.898877 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:51:58.912 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:51:58.912810 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:52:09.016 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:52:09.015571 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:52:09.031 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:52:09.031547 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:52:19.116 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:52:19.116730 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:52:19.130 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:52:19.130876 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:52:29.200 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:52:29.199829 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:52:29.209 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:52:29.209186 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:52:39.327 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:52:39.327433 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:52:39.342 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:52:39.341990 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:52:49.426 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:52:49.426201 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:52:49.435 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:52:49.435635 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:52:59.563 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:52:59.562996 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:52:59.575 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:52:59.575795 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:53:09.688 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:53:09.688135 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:53:09.700 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:53:09.700900 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:53:19.794 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:53:19.793938 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:53:19.808 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:53:19.808242 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:53:29.866 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:53:29.862651 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:53:29.876 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:53:29.876019 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:53:39.949 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:53:39.949273 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:53:40.008 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:53:40.008263 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:53:50.079 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:53:50.079896 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:53:50.091 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:53:50.091883 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:54:00.167 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:54:00.167187 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:54:00.180 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:54:00.180093 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:54:10.275 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:54:10.275419 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:54:10.290 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:54:10.290524 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:54:20.363 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:54:20.363393 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:54:20.449 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:54:20.449156 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:54:30.516 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:54:30.516194 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:54:30.529 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:54:30.529991 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:54:40.603 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:54:40.603135 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:54:40.617 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:54:40.617300 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:54:50.720 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:54:50.720110 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:54:50.733 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:54:50.733108 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:55:00.828 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:55:00.828646 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:55:00.845 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:55:00.843807 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:55:10.917 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:55:10.917253 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:55:10.929 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:55:10.929978 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:55:21.069 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:55:21.068885 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:55:21.092 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:55:21.083544 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:55:31.151 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:55:31.151003 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:55:31.171 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:55:31.171604 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:55:41.271 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:55:41.270860 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:55:41.282 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:55:41.282664 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:55:51.354 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:55:51.354245 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:55:51.366 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:55:51.367036 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:56:01.441 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:56:01.441314 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:56:01.454 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:56:01.454416 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:56:11.572 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:56:11.571760 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:56:11.585 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:56:11.585499 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:56:21.694 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:56:21.693888 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:56:21.707 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:56:21.707005 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:56:31.777 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:56:31.777554 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:56:31.797 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:56:31.792239 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:56:41.864 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:56:41.864017 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:56:41.941 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:56:41.941691 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:56:52.039 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:56:52.039606 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:56:52.054 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:56:52.054540 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:57:02.170 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:57:02.170792 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:57:02.183 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:57:02.183314 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:57:12.297 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:57:12.297170 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:57:12.310 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:57:12.310473 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:57:22.381 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:57:22.381418 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:57:22.401 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:57:22.401951 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:57:32.507 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:57:32.507360 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:57:32.523 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:57:32.523483 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:57:42.628 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:57:42.628438 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:57:42.641 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:57:42.641316 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:57:52.713 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:57:52.713287 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:57:52.725 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:57:52.725570 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:58:02.790 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:58:02.790031 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:58:02.801 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:58:02.801743 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:58:12.875 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:58:12.875275 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:58:12.932 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:58:12.932460 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:58:22.992 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:58:22.992355 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:58:23.006 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:58:23.006346 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:58:33.079 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:58:33.079263 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:58:33.091 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:58:33.091893 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:58:43.184 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:58:43.184542 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:58:43.198 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:58:43.199057 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:58:53.324 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:58:53.324120 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:58:53.338 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:58:53.338259 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:59:03.412 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:59:03.412252 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:59:03.422 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:59:03.423058 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:59:13.511 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:59:13.511906 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:59:13.522 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:59:13.522637 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:59:23.633 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:59:23.633556 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:59:23.644 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:59:23.644332 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:59:33.784 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:59:33.784015 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:59:33.797 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:59:33.797670 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:59:44.048 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:59:44.048366 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:59:44.062 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:59:44.062904 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:59:54.184 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:59:54.184373 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:59:54.195 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 04:59:54.195829 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:00:04.265 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:00:04.265320 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:00:04.367 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:00:04.367427 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:00:14.456 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:00:14.456493 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:00:14.468 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:00:14.469167 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:00:24.599 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:00:24.599393 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:00:24.612 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:00:24.613059 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:00:34.746 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:00:34.743729 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:00:34.753 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:00:34.753956 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:00:44.828 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:00:44.820936 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:00:44.834 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:00:44.834515 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:00:54.969 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:00:54.969545 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:00:54.980 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:00:54.981134 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:01:05.061 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:01:05.061608 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:01:05.072 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:01:05.073262 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:01:15.168 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:01:15.168283 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:01:15.239 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:01:15.238855 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:01:25.312 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:01:25.312546 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:01:25.326 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:01:25.326550 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:01:35.434 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:01:35.434075 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:01:35.449 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:01:35.449427 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:01:45.515 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:01:45.515671 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:01:45.588 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:01:45.588123 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:01:55.716 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:01:55.716090 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:01:55.727 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:01:55.727766 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:02:01.018 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:02:01.018100 I | op-mon: parsing mon endpoints: a=10.100.186.100:6789,b=10.111.132.151:6789,c=10.101.46.100:6789 2022-03-10T05:02:01.687 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:02:01.686586 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T05:02:01.676+0000 7f9f15831380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T05:02:01.687 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10T05:02:01.676+0000 7f9f15831380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T05:02:01.688 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10T05:02:01.676+0000 7f9f15831380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T05:02:01.688 INFO:tasks.rook.operator.smithi037.stdout:couldn't init storage provider. : exit status 5 2022-03-10T05:02:05.852 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:02:05.850710 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:02:05.865 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:02:05.865996 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:02:15.978 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:02:15.978089 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:02:15.991 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:02:15.991404 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:02:26.097 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:02:26.097571 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:02:26.109 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:02:26.110193 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:02:36.223 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:02:36.223414 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:02:36.238 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:02:36.238607 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:02:46.311 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:02:46.311155 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:02:46.324 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:02:46.323778 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:02:56.419 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:02:56.419074 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:02:56.432 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:02:56.432603 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:03:06.508 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:03:06.507913 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:03:06.522 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:03:06.521927 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:03:16.599 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:03:16.599155 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:03:16.614 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:03:16.614059 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:03:26.719 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:03:26.719205 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:03:26.735 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:03:26.735502 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:03:36.814 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:03:36.814144 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:03:36.827 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:03:36.827950 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:03:46.980 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:03:46.980505 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:03:46.993 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:03:46.993751 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:03:57.085 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:03:57.084759 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:03:57.099 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:03:57.099687 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:04:07.172 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:04:07.172471 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:04:07.186 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:04:07.186788 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:04:17.257 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:04:17.256921 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:04:17.271 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:04:17.271944 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:04:27.384 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:04:27.384476 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:04:27.397 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:04:27.398025 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:04:37.507 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:04:37.506746 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:04:37.599 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:04:37.521793 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:04:47.595 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:04:47.594821 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:04:47.609 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:04:47.609348 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:04:57.687 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:04:57.687159 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:04:57.737 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:04:57.737252 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:05:07.809 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:05:07.808917 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:05:07.821 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:05:07.821436 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:05:17.894 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:05:17.894229 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:05:17.905 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:05:17.905869 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:05:27.984 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:05:27.984801 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:05:27.999 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:05:27.999442 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:05:38.088 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:05:38.088506 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:05:38.140 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:05:38.140641 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:05:48.252 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:05:48.252285 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:05:48.263 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:05:48.263499 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:05:58.334 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:05:58.334140 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:05:58.394 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:05:58.394794 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:06:08.468 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:06:08.468333 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:06:08.482 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:06:08.482390 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:06:18.582 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:06:18.582439 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:06:18.594 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:06:18.594831 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:06:28.728 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:06:28.727991 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:06:28.741 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:06:28.741652 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:06:38.846 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:06:38.845749 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:06:38.859 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:06:38.859225 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:06:48.932 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:06:48.931949 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:06:48.946 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:06:48.946480 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:06:59.056 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:06:59.056335 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:06:59.069 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:06:59.069288 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:07:09.182 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:07:09.182822 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:07:09.196 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:07:09.196396 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:07:19.278 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:07:19.277881 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:07:19.288 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:07:19.288802 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:07:29.575 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:07:29.574989 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:07:29.587 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:07:29.587367 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:07:39.735 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:07:39.735858 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:07:39.747 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:07:39.747324 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:07:49.895 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:07:49.895841 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:07:49.904 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:07:49.904797 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:08:00.007 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:08:00.007103 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:08:00.019 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:08:00.019241 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:08:10.088 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:08:10.088450 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:08:10.104 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:08:10.103645 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:08:20.171 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:08:20.171776 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:08:20.184 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:08:20.184728 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:08:30.292 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:08:30.292009 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:08:30.306 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:08:30.306347 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:08:40.409 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:08:40.409156 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:08:40.422 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:08:40.422608 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:08:50.486 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:08:50.486001 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:08:50.499 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:08:50.499383 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:09:00.568 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:09:00.567839 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:09:00.582 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:09:00.582343 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:09:10.711 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:09:10.711638 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:09:10.727 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:09:10.727535 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:09:20.820 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:09:20.820018 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:09:20.833 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:09:20.833269 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:09:30.970 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:09:30.970013 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:09:30.984 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:09:30.984383 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:09:41.052 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:09:41.051927 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:09:41.065 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:09:41.065654 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:09:51.149 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:09:51.149763 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:09:51.162 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:09:51.162277 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:10:01.260 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:10:01.260157 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:10:01.286 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:10:01.286068 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:10:11.404 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:10:11.403850 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:10:11.416 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:10:11.416974 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:10:21.490 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:10:21.490368 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:10:21.543 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:10:21.543410 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:10:31.678 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:10:31.677938 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:10:31.693 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:10:31.692957 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:10:41.801 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:10:41.800782 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:10:41.812 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:10:41.812392 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:10:51.896 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:10:51.896363 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:10:51.909 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:10:51.909514 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:11:02.016 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:11:02.016752 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:11:02.031 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:11:02.031615 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:11:12.135 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:11:12.135491 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:11:12.145 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:11:12.145797 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:11:22.211 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:11:22.211140 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:11:22.222 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:11:22.222697 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:11:32.291 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:11:32.290989 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:11:32.304 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:11:32.304720 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:11:42.378 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:11:42.378007 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:11:42.395 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:11:42.392682 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:11:52.495 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:11:52.495785 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:11:52.512 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:11:52.509808 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:12:02.604 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:12:02.604573 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:12:02.619 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:12:02.619758 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:12:12.687 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:12:12.686903 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:12:12.699 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:12:12.699725 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:12:22.790 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:12:22.790415 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:12:22.845 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:12:22.844844 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:12:32.915 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:12:32.915679 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:12:32.928 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:12:32.928121 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:12:43.039 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:12:43.039084 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:12:43.051 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:12:43.051560 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:12:53.121 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:12:53.121440 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:12:53.135 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:12:53.135986 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:13:03.261 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:13:03.261597 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:13:03.272 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:13:03.272306 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:13:13.369 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:13:13.369056 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:13:13.384 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:13:13.384580 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:13:23.501 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:13:23.501523 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:13:23.516 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:13:23.516443 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:13:33.582 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:13:33.582200 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:13:33.596 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:13:33.596051 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:13:43.674 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:13:43.674324 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:13:43.687 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:13:43.687235 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:13:53.752 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:13:53.752171 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:13:53.765 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:13:53.765043 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:14:03.842 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:14:03.842371 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:14:03.855 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:14:03.855124 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:14:14.019 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:14:14.019699 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:14:14.030 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:14:14.030641 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:14:24.224 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:14:24.224352 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:14:24.238 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:14:24.238831 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:14:34.414 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:14:34.414239 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:14:34.428 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:14:34.428486 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:14:44.518 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:14:44.518142 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:14:44.530 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:14:44.529870 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:14:54.619 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:14:54.619204 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:14:54.633 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:14:54.633230 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:15:04.704 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:15:04.703605 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:15:04.718 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:15:04.717900 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:15:14.841 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:15:14.841044 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:15:14.855 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:15:14.855282 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:15:24.990 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:15:24.990415 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:15:24.999 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:15:24.999751 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:15:35.105 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:15:35.104884 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:15:35.119 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:15:35.119796 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:15:45.228 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:15:45.227771 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:15:45.238 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:15:45.238115 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:15:55.345 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:15:55.345333 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:15:55.362 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:15:55.361950 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:16:05.498 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:16:05.497614 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:16:05.509 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:16:05.509266 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:16:15.621 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:16:15.620733 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:16:15.633 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:16:15.633220 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:16:25.702 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:16:25.701840 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:16:25.783 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:16:25.782592 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:16:35.913 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:16:35.913364 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:16:35.924 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:16:35.924943 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:16:46.038 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:16:46.038476 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:16:46.051 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:16:46.051748 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:16:56.133 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:16:56.133547 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:16:56.186 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:16:56.186426 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:17:06.271 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:17:06.270965 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:17:06.281 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:17:06.281658 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:17:16.346 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:17:16.345605 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:17:16.390 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:17:16.390300 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:17:26.470 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:17:26.469836 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:17:26.522 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:17:26.522415 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:17:36.578 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:17:36.577949 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:17:36.594 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:17:36.593812 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:17:46.695 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:17:46.694839 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:17:46.709 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:17:46.709484 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:17:56.875 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:17:56.874825 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:17:56.888 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:17:56.888321 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:18:06.986 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:18:06.985863 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:18:06.997 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:18:06.997346 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:18:17.107 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:18:17.106515 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:18:17.118 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:18:17.118405 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:18:27.219 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:18:27.218539 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:18:27.232 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:18:27.232699 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:18:37.301 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:18:37.301120 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:18:37.314 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:18:37.313890 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:18:41.702 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:18:41.702429 I | op-mon: parsing mon endpoints: a=10.100.186.100:6789,b=10.111.132.151:6789,c=10.101.46.100:6789 2022-03-10T05:18:42.351 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:18:42.350623 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T05:18:42.336+0000 7fa9007ec380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T05:18:42.351 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10T05:18:42.336+0000 7fa9007ec380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T05:18:42.351 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10T05:18:42.336+0000 7fa9007ec380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T05:18:42.352 INFO:tasks.rook.operator.smithi037.stdout:couldn't init storage provider. : exit status 5 2022-03-10T05:18:47.418 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:18:47.418646 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:18:47.433 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:18:47.433117 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:18:57.515 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:18:57.515322 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:18:57.528 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:18:57.528913 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:19:07.594 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:19:07.593860 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:19:07.606 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:19:07.606548 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:19:17.671 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:19:17.671751 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:19:17.686 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:19:17.687023 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:19:27.799 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:19:27.798559 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:19:27.809 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:19:27.809217 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:19:37.940 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:19:37.939863 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:19:37.949 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:19:37.949563 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:19:48.064 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:19:48.064002 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:19:48.078 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:19:48.078410 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:19:58.210 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:19:58.210254 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:19:58.220 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:19:58.220469 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:20:08.283 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:20:08.283721 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:20:08.296 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:20:08.296191 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:20:18.367 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:20:18.367432 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:20:18.422 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:20:18.422053 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:20:28.493 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:20:28.493355 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:20:28.506 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:20:28.506969 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:20:38.626 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:20:38.626022 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:20:38.638 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:20:38.637983 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:20:48.727 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:20:48.727069 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:20:48.739 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:20:48.739280 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:20:58.838 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:20:58.837770 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:20:58.851 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:20:58.851428 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:21:08.926 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:21:08.925603 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:21:08.940 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:21:08.940698 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:21:19.018 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:21:19.018807 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:21:19.032 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:21:19.032362 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:21:29.219 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:21:29.218747 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:21:29.232 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:21:29.231835 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:21:39.312 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:21:39.312322 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:21:39.385 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:21:39.385646 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:21:49.496 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:21:49.495688 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:21:49.510 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:21:49.510322 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:21:59.579 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:21:59.579651 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:21:59.593 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:21:59.593000 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:22:09.691 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:22:09.690734 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:22:09.704 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:22:09.705081 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:22:19.820 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:22:19.820701 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:22:19.833 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:22:19.833496 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:22:29.989 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:22:29.985104 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:22:30.000 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:22:30.000179 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:22:40.155 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:22:40.155171 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:22:40.171 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:22:40.171048 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:22:50.268 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:22:50.268340 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:22:50.282 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:22:50.282783 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:23:00.347 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:23:00.347356 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:23:00.363 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:23:00.362774 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:23:10.452 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:23:10.451849 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:23:10.465 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:23:10.465661 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:23:20.543 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:23:20.543167 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:23:20.555 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:23:20.555413 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:23:30.655 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:23:30.655303 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:23:30.667 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:23:30.667622 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:23:40.771 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:23:40.771141 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:23:40.787 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:23:40.787064 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:23:50.879 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:23:50.879070 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:23:50.893 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:23:50.893829 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:24:01.009 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:24:01.008941 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:24:01.020 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:24:01.020617 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:24:11.135 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:24:11.135758 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:24:11.149 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:24:11.149973 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:24:21.223 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:24:21.223175 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:24:21.233 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:24:21.233825 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:24:31.313 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:24:31.312934 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:24:31.374 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:24:31.374534 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:24:41.464 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:24:41.463955 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:24:41.479 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:24:41.479225 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:24:51.611 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:24:51.610837 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:24:51.624 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:24:51.624253 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:25:01.729 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:25:01.729259 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:25:01.742 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:25:01.743046 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:25:11.844 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:25:11.843738 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:25:11.859 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:25:11.859424 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:25:21.995 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:25:21.995103 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:25:22.009 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:25:22.009904 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:25:32.132 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:25:32.132776 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:25:32.143 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:25:32.143463 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:25:42.218 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:25:42.218220 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:25:42.228 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:25:42.228928 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:25:52.354 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:25:52.354242 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:25:52.369 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:25:52.369114 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:26:02.484 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:26:02.484021 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:26:02.498 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:26:02.498252 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:26:12.613 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:26:12.612662 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:26:12.625 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:26:12.625126 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:26:22.700 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:26:22.700522 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:26:22.772 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:26:22.772640 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:26:32.843 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:26:32.843397 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:26:32.855 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:26:32.855646 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:26:42.969 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:26:42.969624 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:26:42.981 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:26:42.981870 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:26:53.112 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:26:53.111996 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:26:53.124 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:26:53.124601 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:27:03.250 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:27:03.250294 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:27:03.264 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:27:03.264439 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:27:13.392 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:27:13.392389 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:27:13.403 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:27:13.403510 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:27:23.506 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:27:23.506404 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:27:23.524 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:27:23.524883 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:27:33.619 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:27:33.618902 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:27:33.636 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:27:33.637011 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:27:43.761 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:27:43.761689 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:27:43.777 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:27:43.777096 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:27:53.840 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:27:53.840480 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:27:53.853 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:27:53.853925 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:28:03.928 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:28:03.928269 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:28:03.941 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:28:03.941684 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:28:14.049 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:28:14.048718 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:28:14.065 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:28:14.061600 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:28:24.155 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:28:24.155423 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:28:24.169 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:28:24.170040 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:28:34.240 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:28:34.240251 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:28:34.311 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:28:34.311566 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:28:44.405 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:28:44.405050 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:28:44.417 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:28:44.417985 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:28:54.615 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:28:54.614840 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:28:54.629 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:28:54.629844 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:29:04.694 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:29:04.694478 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:29:04.767 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:29:04.767273 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:29:14.882 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:29:14.882460 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:29:14.898 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:29:14.898717 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:29:25.016 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:29:25.016017 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:29:25.031 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:29:25.031781 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:29:35.137 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:29:35.137311 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:29:35.148 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:29:35.148744 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:29:45.270 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:29:45.269922 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:29:45.283 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:29:45.283222 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:29:55.352 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:29:55.352192 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:29:55.367 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:29:55.367866 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:30:05.477 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:30:05.476686 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:30:05.491 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:30:05.491528 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:30:15.622 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:30:15.622510 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:30:15.634 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:30:15.634592 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:30:25.766 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:30:25.766172 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:30:25.779 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:30:25.779717 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:30:35.904 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:30:35.904039 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:30:35.914 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:30:35.914964 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:30:46.044 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:30:46.043770 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:30:46.057 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:30:46.057329 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:30:56.144 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:30:56.144441 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:30:56.158 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:30:56.158488 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:31:06.253 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:31:06.252985 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:31:06.265 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:31:06.265913 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:31:16.372 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:31:16.368205 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:31:16.389 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:31:16.389870 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:31:26.528 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:31:26.528047 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:31:26.539 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:31:26.539086 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:31:36.682 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:31:36.681775 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:31:36.757 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:31:36.757105 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:31:46.834 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:31:46.834113 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:31:46.846 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:31:46.846015 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:31:56.948 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:31:56.947921 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:31:56.959 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:31:56.960171 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:32:07.026 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:32:07.026253 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:32:07.041 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:32:07.040954 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:32:17.114 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:32:17.114528 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:32:17.128 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:32:17.128410 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:32:27.206 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:32:27.206685 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:32:27.218 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:32:27.219003 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:32:37.287 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:32:37.287336 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:32:37.300 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:32:37.300977 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:32:47.363 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:32:47.363718 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:32:47.376 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:32:47.376391 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:32:57.471 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:32:57.471622 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:32:57.485 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:32:57.485157 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:33:07.552 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:33:07.552252 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:33:07.601 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:33:07.600993 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:33:17.673 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:33:17.673822 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:33:17.687 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:33:17.687608 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:33:27.803 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:33:27.802595 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:33:27.815 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:33:27.815318 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:33:37.918 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:33:37.918289 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:33:37.934 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:33:37.934390 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:33:48.045 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:33:48.045186 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:33:48.057 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:33:48.057616 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:33:58.140 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:33:58.139957 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:33:58.154 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:33:58.154516 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:34:08.259 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:34:08.259194 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:34:08.272 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:34:08.272183 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:34:18.340 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:34:18.339939 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:34:18.353 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:34:18.353717 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:34:28.426 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:34:28.426179 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:34:28.437 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:34:28.437862 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:34:38.512 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:34:38.512362 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:34:38.525 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:34:38.526092 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:34:48.590 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:34:48.590488 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:34:48.604 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:34:48.605034 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:34:58.676 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:34:58.676134 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:34:58.688 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:34:58.688772 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:35:08.808 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:35:08.807862 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:35:08.821 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:35:08.821304 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:35:18.897 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:35:18.897278 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:35:18.960 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:35:18.960948 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:35:22.357 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:35:22.357424 I | op-mon: parsing mon endpoints: a=10.100.186.100:6789,b=10.111.132.151:6789,c=10.101.46.100:6789 2022-03-10T05:35:23.048 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:35:23.047704 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T05:35:23.037+0000 7f07c84f0380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T05:35:23.048 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10T05:35:23.037+0000 7f07c84f0380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T05:35:23.048 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10T05:35:23.037+0000 7f07c84f0380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T05:35:23.049 INFO:tasks.rook.operator.smithi037.stdout:couldn't init storage provider. : exit status 5 2022-03-10T05:35:29.101 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:35:29.101506 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:35:29.116 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:35:29.115270 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:35:39.195 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:35:39.194876 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:35:39.208 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:35:39.208549 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:35:49.276 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:35:49.276645 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:35:49.291 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:35:49.291553 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:35:59.435 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:35:59.435498 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:35:59.447 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:35:59.447914 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:36:09.512 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:36:09.512506 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:36:09.525 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:36:09.525776 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:36:19.629 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:36:19.629212 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:36:19.645 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:36:19.645441 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:36:29.742 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:36:29.742100 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:36:29.753 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:36:29.753796 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:36:39.863 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:36:39.863386 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:36:39.877 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:36:39.878006 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:36:49.979 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:36:49.978997 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:36:49.991 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:36:49.991501 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:37:00.121 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:37:00.120723 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:37:00.135 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:37:00.134896 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:37:10.264 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:37:10.264208 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:37:10.274 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:37:10.275277 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:37:20.437 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:37:20.437346 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:37:20.452 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:37:20.452801 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:37:30.519 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:37:30.519274 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:37:30.531 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:37:30.531742 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:37:40.604 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:37:40.603874 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:37:40.616 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:37:40.616501 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:37:50.763 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:37:50.763286 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:37:50.775 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:37:50.775957 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:38:00.888 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:38:00.888195 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:38:00.900 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:38:00.900700 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:38:10.991 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:38:10.991418 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:38:11.010 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:38:11.010514 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:38:21.088 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:38:21.088445 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:38:21.101 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:38:21.101098 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:38:31.213 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:38:31.212939 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:38:31.221 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:38:31.222284 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:38:41.290 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:38:41.290797 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:38:41.301 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:38:41.301376 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:38:51.427 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:38:51.427064 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:38:51.440 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:38:51.438709 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:39:01.581 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:39:01.581665 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:39:01.592 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:39:01.593151 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:39:11.658 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:39:11.658274 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:39:11.673 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:39:11.673784 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:39:21.771 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:39:21.771055 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:39:21.785 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:39:21.786209 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:39:31.859 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:39:31.859148 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:39:31.912 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:39:31.910886 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:39:42.042 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:39:42.041936 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:39:42.058 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:39:42.058641 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:39:52.175 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:39:52.174743 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:39:52.187 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:39:52.188133 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:40:02.296 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:40:02.296269 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:40:02.312 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:40:02.312969 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:40:12.425 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:40:12.424859 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:40:12.439 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:40:12.439615 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:40:22.519 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:40:22.518970 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:40:22.579 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:40:22.579327 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:40:32.644 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:40:32.644398 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:40:32.695 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:40:32.695149 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:40:42.769 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:40:42.769234 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:40:42.846 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:40:42.846015 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:40:52.922 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:40:52.921971 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:40:52.941 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:40:52.936859 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:41:03.013 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:41:03.013007 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:41:03.024 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:41:03.024643 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:41:13.106 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:41:13.106338 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:41:13.118 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:41:13.119260 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:41:23.202 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:41:23.202134 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:41:23.214 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:41:23.214670 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:41:33.286 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:41:33.286764 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:41:33.300 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:41:33.300963 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:41:43.403 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:41:43.403487 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:41:43.415 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:41:43.415595 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:41:53.473 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:41:53.473742 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:41:53.487 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:41:53.487562 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:42:03.580 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:42:03.580841 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:42:03.597 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:42:03.597166 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:42:13.720 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:42:13.720381 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:42:13.732 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:42:13.732804 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:42:23.851 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:42:23.851479 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:42:23.863 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:42:23.864221 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:42:33.978 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:42:33.978550 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:42:33.991 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:42:33.991903 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:42:44.087 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:42:44.087301 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:42:44.102 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:42:44.102734 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:42:54.215 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:42:54.215232 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:42:54.227 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:42:54.227485 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:43:04.342 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:43:04.342097 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:43:04.358 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:43:04.358478 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:43:14.518 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:43:14.518080 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:43:14.535 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:43:14.535980 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:43:24.650 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:43:24.650038 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:43:24.661 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:43:24.661635 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:43:34.808 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:43:34.808562 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:43:34.823 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:43:34.823752 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:43:44.885 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:43:44.885479 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:43:44.901 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:43:44.901546 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:43:55.036 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:43:55.035226 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:43:55.047 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:43:55.047671 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:44:05.139 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:44:05.138293 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:44:05.189 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:44:05.189311 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:44:15.289 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:44:15.289282 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:44:15.303 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:44:15.302781 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:44:25.421 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:44:25.421407 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:44:25.437 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:44:25.437213 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:44:35.574 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:44:35.574398 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:44:35.585 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:44:35.586040 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:44:45.886 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:44:45.886090 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:44:45.897 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:44:45.897879 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:44:55.966 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:44:55.956091 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:44:55.968 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:44:55.967823 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:45:06.045 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:45:06.045562 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:45:06.059 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:45:06.059948 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:45:16.145 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:45:16.145509 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:45:16.209 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:45:16.209111 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:45:26.308 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:45:26.308058 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:45:26.321 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:45:26.321935 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:45:36.424 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:45:36.424386 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:45:36.437 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:45:36.436767 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:45:46.508 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:45:46.508248 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:45:46.634 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:45:46.559335 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:45:56.683 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:45:56.683803 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:45:56.696 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:45:56.696643 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:46:06.782 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:46:06.782698 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:46:06.826 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:46:06.826330 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:46:16.898 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:46:16.898810 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:46:16.911 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:46:16.911978 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:46:26.974 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:46:26.973075 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:46:26.982 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:46:26.982665 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:46:37.061 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:46:37.060769 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:46:37.075 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:46:37.075697 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:46:47.135 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:46:47.135407 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:46:47.144 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:46:47.144205 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:46:57.220 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:46:57.220356 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:46:57.235 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:46:57.235231 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:47:07.353 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:47:07.352643 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:47:07.364 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:47:07.364984 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:47:17.524 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:47:17.524110 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:47:17.554 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:47:17.554033 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:47:27.685 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:47:27.685123 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:47:27.695 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:47:27.695645 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:47:37.802 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:47:37.802581 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:47:37.815 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:47:37.815030 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:47:47.879 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:47:47.878693 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:47:47.894 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:47:47.894850 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:47:57.993 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:47:57.991906 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:47:58.007 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:47:58.006967 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:48:08.078 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:48:08.077830 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:48:08.091 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:48:08.091306 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:48:18.160 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:48:18.160410 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:48:18.171 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:48:18.171227 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:48:28.246 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:48:28.245846 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:48:28.257 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:48:28.257174 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:48:38.324 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:48:38.324504 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:48:38.336 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:48:38.336674 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:48:48.443 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:48:48.442950 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:48:48.457 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:48:48.458075 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:48:58.539 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:48:58.539315 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:48:58.593 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:48:58.593235 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:49:08.708 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:49:08.708043 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:49:08.739 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:49:08.739420 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:49:18.867 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:49:18.867356 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:49:18.881 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:49:18.881576 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:49:28.964 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:49:28.964590 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:49:28.977 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:49:28.977573 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:49:39.084 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:49:39.084758 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:49:39.097 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:49:39.098347 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:49:49.245 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:49:49.245268 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:49:49.257 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:49:49.257707 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:49:59.437 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:49:59.438300 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:49:59.446 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:49:59.447685 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:50:09.524 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:50:09.524994 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:50:09.542 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:50:09.543330 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:50:19.677 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:50:19.677796 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:50:19.692 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:50:19.693173 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:50:29.786 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:50:29.786617 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:50:29.799 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:50:29.800749 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:50:39.926 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:50:39.927276 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:50:39.939 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:50:39.940590 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:50:50.014 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:50:50.015174 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:50:50.027 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:50:50.028808 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:51:00.102 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:51:00.103129 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:51:00.114 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:51:00.116072 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:51:10.191 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:51:10.192041 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:51:10.201 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:51:10.203168 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:51:20.321 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:51:20.322734 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:51:20.334 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:51:20.336271 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:51:30.410 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:51:30.411518 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:51:30.423 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:51:30.425136 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:51:40.542 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:51:40.543447 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:51:40.557 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:51:40.558871 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:51:50.693 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:51:50.694477 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:51:50.707 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:51:50.708618 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:52:00.802 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:52:00.803968 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:52:00.877 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:52:00.879068 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:52:03.054 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:52:03.055953 I | op-mon: parsing mon endpoints: a=10.100.186.100:6789,b=10.111.132.151:6789,c=10.101.46.100:6789 2022-03-10T05:52:03.754 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:52:03.755667 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T05:52:03.745+0000 7f4e701d4380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T05:52:03.755 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10T05:52:03.745+0000 7f4e701d4380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T05:52:03.755 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10T05:52:03.745+0000 7f4e701d4380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T05:52:03.755 INFO:tasks.rook.operator.smithi037.stdout:couldn't init storage provider. : exit status 5 2022-03-10T05:52:10.992 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:52:10.994269 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:52:11.003 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:52:11.004785 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:52:21.112 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:52:21.113958 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:52:21.125 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:52:21.127258 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:52:31.234 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:52:31.236228 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:52:31.338 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:52:31.247706 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:52:41.424 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:52:41.426058 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:52:41.438 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:52:41.440193 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:52:51.518 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:52:51.520271 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:52:51.527 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:52:51.529206 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:53:01.629 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:53:01.631390 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:53:01.645 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:53:01.648043 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:53:11.777 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:53:11.779416 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:53:11.789 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:53:11.791910 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:53:21.921 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:53:21.923361 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:53:21.960 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:53:21.962158 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:53:32.028 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:53:32.030698 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:53:32.043 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:53:32.045969 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:53:42.143 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:53:42.145329 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:53:42.156 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:53:42.159447 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:53:52.302 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:53:52.305054 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:53:52.311 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:53:52.314499 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:54:02.407 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:54:02.409198 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:54:02.424 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:54:02.427064 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:54:12.514 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:54:12.516788 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:54:12.527 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:54:12.530119 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:54:22.619 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:54:22.621509 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:54:22.637 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:54:22.640138 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:54:32.741 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:54:32.743468 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:54:32.754 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:54:32.757039 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:54:42.842 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:54:42.844429 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:54:42.854 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:54:42.857597 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:54:52.962 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:54:52.964605 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:54:52.974 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:54:52.976626 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:55:03.064 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:55:03.067342 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:55:03.078 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:55:03.081711 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:55:13.166 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:55:13.169530 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:55:13.178 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:55:13.181909 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:55:23.287 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:55:23.290254 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:55:23.299 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:55:23.302831 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:55:33.365 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:55:33.368656 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:55:33.381 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:55:33.383907 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:55:43.475 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:55:43.477847 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:55:43.485 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:55:43.488295 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:55:53.543 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:55:53.542118 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:55:53.551 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:55:53.554075 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:56:03.663 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:56:03.666537 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:56:03.677 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:56:03.680699 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:56:13.748 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:56:13.751581 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:56:13.761 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:56:13.764338 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:56:23.837 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:56:23.840470 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:56:23.850 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:56:23.853161 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:56:33.958 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:56:33.961931 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:56:33.972 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:56:33.975654 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:56:44.091 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:56:44.094442 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:56:44.104 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:56:44.107557 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:56:54.248 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:56:54.251326 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:56:54.263 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:56:54.266634 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:57:04.392 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:57:04.395396 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:57:04.402 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:57:04.406543 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:57:14.470 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:57:14.473707 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:57:14.480 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:57:14.484173 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:57:24.546 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:57:24.546682 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:57:24.554 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:57:24.557094 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:57:34.624 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:57:34.627537 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:57:34.635 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:57:34.639319 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:57:44.765 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:57:44.768351 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:57:44.776 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:57:44.780174 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:57:54.893 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:57:54.896139 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:57:54.903 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:57:54.907225 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:58:05.001 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:58:05.005080 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:58:05.057 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:58:05.060580 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:58:15.121 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:58:15.123818 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:58:15.132 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:58:15.136195 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:58:25.242 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:58:25.245771 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:58:25.255 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:58:25.259438 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:58:35.327 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:58:35.329970 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:58:35.338 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:58:35.341957 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:58:45.457 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:58:45.460906 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:58:45.472 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:58:45.476206 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:58:55.585 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:58:55.588898 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:58:55.599 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:58:55.603510 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:59:05.715 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:59:05.718695 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:59:05.727 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:59:05.731191 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:59:15.855 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:59:15.859862 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:59:15.867 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:59:15.871471 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:59:25.939 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:59:25.942982 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:59:25.951 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:59:25.955326 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:59:36.075 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:59:36.078589 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:59:36.088 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:59:36.091957 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:59:46.192 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:59:46.196403 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:59:46.205 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:59:46.209731 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:59:56.312 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:59:56.316022 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:59:56.324 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 05:59:56.328775 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:00:06.408 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:00:06.411223 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:00:06.419 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:00:06.423351 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:00:16.537 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:00:16.541580 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:00:16.552 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:00:16.555925 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:00:26.690 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:00:26.694494 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:00:26.702 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:00:26.707172 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:00:36.784 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:00:36.788290 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:00:36.796 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:00:36.800569 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:00:46.877 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:00:46.881516 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:00:46.891 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:00:46.895214 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:00:56.970 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:00:56.974565 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:00:56.985 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:00:56.989289 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:01:07.092 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:01:07.096694 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:01:07.105 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:01:07.110156 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:01:17.173 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:01:17.177159 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:01:17.241 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:01:17.245436 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:01:27.341 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:01:27.344947 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:01:27.354 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:01:27.359342 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:01:37.428 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:01:37.432386 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:01:37.466 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:01:37.470670 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:01:47.581 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:01:47.585693 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:01:47.594 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:01:47.599068 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:01:57.694 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:01:57.699147 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:01:57.710 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:01:57.713876 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:02:07.831 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:02:07.835388 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:02:07.843 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:02:07.847963 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:02:17.968 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:02:17.969389 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:02:17.978 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:02:17.982901 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:02:28.084 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:02:28.088774 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:02:28.098 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:02:28.103077 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:02:38.288 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:02:38.292341 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:02:38.296 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:02:38.301618 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:02:48.371 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:02:48.375731 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:02:48.384 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:02:48.388711 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:02:58.495 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:02:58.500498 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:02:58.508 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:02:58.513132 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:03:08.615 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:03:08.619745 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:03:08.627 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:03:08.632197 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:03:18.711 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:03:18.715636 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:03:18.724 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:03:18.728846 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:03:28.852 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:03:28.857371 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:03:28.862 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:03:28.867364 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:03:38.964 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:03:38.969194 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:03:38.980 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:03:38.984952 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:03:49.081 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:03:49.086068 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:03:49.095 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:03:49.100294 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:03:59.319 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:03:59.323631 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:03:59.359 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:03:59.364754 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:04:09.451 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:04:09.456298 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:04:09.465 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:04:09.470704 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:04:19.632 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:04:19.636792 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:04:19.645 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:04:19.650100 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:04:29.743 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:04:29.747782 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:04:29.753 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:04:29.758039 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:04:39.821 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:04:39.826737 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:04:39.836 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:04:39.841621 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:04:49.959 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:04:49.963941 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:04:49.973 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:04:49.978187 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:05:00.046 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:05:00.051017 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:05:00.062 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:05:00.067464 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:05:10.189 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:05:10.194138 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:05:10.201 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:05:10.206433 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:05:20.307 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:05:20.311410 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:05:20.320 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:05:20.325800 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:05:30.387 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:05:30.392555 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:05:30.435 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:05:30.440484 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:05:40.544 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:05:40.549740 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:05:40.557 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:05:40.562478 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:05:50.625 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:05:50.630799 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:05:50.676 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:05:50.681213 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:06:00.791 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:06:00.796267 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:06:00.807 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:06:00.812987 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:06:10.920 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:06:10.925854 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:06:10.936 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:06:10.941447 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:06:21.010 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:06:21.015193 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:06:21.024 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:06:21.029500 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:06:31.134 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:06:31.139737 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:06:31.145 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:06:31.151258 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:06:41.209 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:06:41.215090 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:06:41.223 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:06:41.229086 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:06:51.348 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:06:51.353997 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:06:51.365 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:06:51.370808 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:07:01.475 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:07:01.480018 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:07:01.488 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:07:01.493440 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:07:11.606 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:07:11.612017 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:07:11.622 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:07:11.627867 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:07:21.686 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:07:21.691976 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:07:21.705 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:07:21.710487 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:07:31.801 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:07:31.806354 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:07:31.815 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:07:31.821109 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:07:42.002 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:07:42.007337 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:07:42.017 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:07:42.023041 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:07:52.135 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:07:52.140601 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:07:52.149 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:07:52.154845 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:08:02.265 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:08:02.270427 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:08:02.276 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:08:02.282500 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:08:12.402 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:08:12.407443 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:08:12.429 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:08:12.434417 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:08:22.526 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:08:22.531369 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:08:22.537 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:08:22.543043 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:08:32.642 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:08:32.647661 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:08:32.656 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:08:32.662173 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:08:42.724 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:08:42.728684 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:08:42.736 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:08:42.742055 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:08:43.762 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:08:43.767574 I | op-mon: parsing mon endpoints: a=10.100.186.100:6789,b=10.111.132.151:6789,c=10.101.46.100:6789 2022-03-10T06:08:44.439 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:08:44.444606 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T06:08:44.433+0000 7f09847f7380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T06:08:44.440 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10T06:08:44.433+0000 7f09847f7380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T06:08:44.440 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10T06:08:44.433+0000 7f09847f7380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T06:08:44.440 INFO:tasks.rook.operator.smithi037.stdout:couldn't init storage provider. : exit status 5 2022-03-10T06:08:52.804 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:08:52.809234 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:08:52.817 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:08:52.822633 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:09:02.894 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:09:02.899354 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:09:02.905 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:09:02.910828 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:09:13.003 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:09:13.008498 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:09:13.016 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:09:13.021707 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:09:23.121 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:09:23.126443 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:09:23.134 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:09:23.139629 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:09:33.230 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:09:33.235807 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:09:33.246 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:09:33.252102 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:09:43.401 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:09:43.406725 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:09:43.413 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:09:43.418578 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:09:53.511 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:09:53.516749 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:09:53.523 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:09:53.528916 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:10:03.632 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:10:03.636413 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:10:03.646 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:10:03.652165 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:10:13.717 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:10:13.723085 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:10:13.729 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:10:13.735314 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:10:23.829 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:10:23.835278 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:10:23.845 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:10:23.851058 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:10:33.914 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:10:33.920084 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:10:33.932 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:10:33.933770 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:10:44.001 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:10:44.007372 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:10:44.015 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:10:44.021218 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:10:54.090 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:10:54.093975 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:10:54.103 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:10:54.109154 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:11:04.192 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:11:04.197984 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:11:04.206 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:11:04.212968 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:11:14.272 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:11:14.278042 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:11:14.284 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:11:14.290440 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:11:24.358 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:11:24.362731 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:11:24.415 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:11:24.420691 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:11:34.525 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:11:34.530648 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:11:34.539 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:11:34.545247 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:11:44.620 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:11:44.626039 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:11:44.633 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:11:44.639007 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:11:54.728 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:11:54.733879 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:11:54.741 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:11:54.747773 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:12:04.864 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:12:04.870060 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:12:04.876 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:12:04.882389 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:12:14.957 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:12:14.963290 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:12:15.028 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:12:15.034080 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:12:25.132 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:12:25.138260 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:12:25.145 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:12:25.151142 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:12:35.254 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:12:35.260565 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:12:35.269 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:12:35.275360 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:12:45.342 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:12:45.348682 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:12:45.355 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:12:45.361781 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:12:55.453 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:12:55.459655 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:12:55.470 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:12:55.475661 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:13:05.577 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:13:05.583155 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:13:05.590 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:13:05.596896 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:13:15.661 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:13:15.667460 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:13:15.676 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:13:15.682773 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:13:25.774 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:13:25.779841 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:13:25.785 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:13:25.792103 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:13:35.901 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:13:35.907203 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:13:35.912 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:13:35.918147 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:13:46.011 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:13:46.017736 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:13:46.030 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:13:46.034835 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:13:56.135 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:13:56.141921 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:13:56.146 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:13:56.152909 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:14:06.239 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:14:06.243773 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:14:06.253 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:14:06.259412 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:14:16.332 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:14:16.338509 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:14:16.346 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:14:16.352225 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:14:26.429 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:14:26.434960 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:14:26.438 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:14:26.444678 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:14:36.509 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:14:36.515392 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:14:36.523 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:14:36.529142 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:14:46.623 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:14:46.629737 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:14:46.633 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:14:46.639814 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:14:56.701 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:14:56.707006 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:14:56.713 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:14:56.719333 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:15:06.811 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:15:06.817664 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:15:06.824 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:15:06.830213 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:15:16.900 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:15:16.906224 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:15:16.926 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:15:16.927492 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:15:27.021 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:15:27.027133 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:15:27.033 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:15:27.039943 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:15:37.120 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:15:37.126181 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:15:37.132 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:15:37.139128 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:15:47.214 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:15:47.220734 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:15:47.228 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:15:47.235186 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:15:57.329 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:15:57.335440 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:15:57.343 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:15:57.349758 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:16:07.446 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:16:07.452680 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:16:07.459 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:16:07.465413 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:16:17.531 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:16:17.537847 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:16:17.573 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:16:17.579974 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:16:27.698 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:16:27.704703 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:16:27.713 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:16:27.719591 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:16:37.780 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:16:37.786540 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:16:37.794 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:16:37.800979 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:16:47.866 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:16:47.872088 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:16:47.878 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:16:47.884709 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:16:57.953 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:16:57.959820 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:16:57.995 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:16:58.001451 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:17:08.117 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:17:08.124014 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:17:08.129 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:17:08.135982 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:17:18.228 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:17:18.234948 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:17:18.240 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:17:18.246941 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:17:28.356 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:17:28.362110 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:17:28.367 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:17:28.374367 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:17:38.447 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:17:38.453211 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:17:38.458 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:17:38.465256 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:17:48.529 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:17:48.535729 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:17:48.584 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:17:48.590611 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:17:58.677 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:17:58.683324 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:17:58.691 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:17:58.698295 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:18:08.767 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:18:08.773331 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:18:08.780 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:18:08.786414 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:18:18.916 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:18:18.922729 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:18:18.928 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:18:18.934737 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:18:28.995 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:18:29.001921 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:18:29.010 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:18:29.016722 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:18:39.112 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:18:39.118123 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:18:39.124 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:18:39.130762 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:18:49.189 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:18:49.195638 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:18:49.202 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:18:49.208597 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:18:59.307 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:18:59.312573 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:18:59.317 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:18:59.323314 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:19:09.393 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:19:09.400152 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:19:09.407 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:19:09.414088 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:19:19.517 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:19:19.523603 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:19:19.532 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:19:19.539374 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:19:29.598 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:19:29.604686 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:19:29.611 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:19:29.618284 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:19:39.726 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:19:39.732464 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:19:39.743 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:19:39.749621 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:19:49.820 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:19:49.826967 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:19:49.835 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:19:49.842534 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:19:59.905 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:19:59.911844 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:19:59.919 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:19:59.926467 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:20:09.990 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:20:09.996803 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:20:10.005 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:20:10.012189 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:20:20.110 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:20:20.116578 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:20:20.124 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:20:20.131291 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:20:30.240 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:20:30.246578 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:20:30.251 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:20:30.258229 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:20:40.351 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:20:40.356581 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:20:40.360 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:20:40.366716 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:20:50.479 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:20:50.485588 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:20:50.494 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:20:50.501059 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:21:00.613 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:21:00.620078 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:21:00.626 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:21:00.632909 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:21:10.740 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:21:10.746334 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:21:10.753 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:21:10.759933 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:21:20.919 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:21:20.925942 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:21:20.932 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:21:20.939323 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:21:31.045 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:21:31.051774 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:21:31.058 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:21:31.064896 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:21:41.145 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:21:41.151325 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:21:41.156 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:21:41.162851 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:21:51.310 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:21:51.317159 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:21:51.323 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:21:51.330118 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:22:01.440 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:22:01.446942 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:22:01.454 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:22:01.460936 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:22:11.555 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:22:11.561953 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:22:11.570 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:22:11.576892 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:22:21.685 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:22:21.691305 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:22:21.700 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:22:21.707047 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:22:31.792 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:22:31.799212 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:22:31.807 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:22:31.814354 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:22:41.922 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:22:41.928353 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:22:41.935 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:22:41.942311 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:22:52.038 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:22:52.044637 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:22:52.052 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:22:52.058107 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:23:02.137 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:23:02.143954 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:23:02.147 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:23:02.154274 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:23:12.207 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:23:12.213497 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:23:12.220 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:23:12.227485 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:23:22.296 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:23:22.302630 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:23:22.309 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:23:22.316150 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:23:32.419 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:23:32.425473 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:23:32.434 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:23:32.439981 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:23:42.507 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:23:42.513594 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:23:42.523 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:23:42.529393 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:23:52.656 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:23:52.661845 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:23:52.669 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:23:52.675335 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:24:02.740 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:24:02.746210 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:24:02.752 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:24:02.758709 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:24:12.820 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:24:12.825727 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:24:12.869 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:24:12.875176 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:24:22.976 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:24:22.981908 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:24:22.991 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:24:22.997344 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:24:33.091 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:24:33.096059 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:24:33.106 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:24:33.110249 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:24:43.223 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:24:43.228895 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:24:43.238 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:24:43.243122 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:24:53.356 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:24:53.361604 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:24:53.367 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:24:53.372210 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:25:03.454 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:25:03.459316 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:25:03.466 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:25:03.471673 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:25:13.539 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:25:13.544410 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:25:13.560 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:25:13.565232 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:25:23.670 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:25:23.674863 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:25:23.683 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:25:23.688203 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:25:24.449 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:25:24.453698 I | op-mon: parsing mon endpoints: a=10.100.186.100:6789,b=10.111.132.151:6789,c=10.101.46.100:6789 2022-03-10T06:25:25.203 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:25:25.207428 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T06:25:25.197+0000 7fa49d973380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T06:25:25.203 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10T06:25:25.197+0000 7fa49d973380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T06:25:25.204 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10T06:25:25.197+0000 7fa49d973380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T06:25:25.204 INFO:tasks.rook.operator.smithi037.stdout:couldn't init storage provider. : exit status 5 2022-03-10T06:25:33.755 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:25:33.759550 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:25:33.767 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:25:33.772154 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:25:43.848 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:25:43.852756 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:25:43.861 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:25:43.866260 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:25:53.943 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:25:53.947913 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:25:53.956 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:25:53.960296 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:26:04.089 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:26:04.093764 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:26:04.125 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:26:04.129594 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:26:14.256 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:26:14.260136 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:26:14.267 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:26:14.271101 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:26:24.331 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:26:24.335217 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:26:24.387 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:26:24.391140 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:26:34.489 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:26:34.493096 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:26:34.503 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:26:34.507525 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:26:44.615 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:26:44.618482 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:26:44.627 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:26:44.631223 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:26:54.693 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:26:54.696995 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:26:54.709 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:26:54.712859 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:27:04.780 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:27:04.783731 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:27:04.835 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:27:04.838578 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:27:14.911 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:27:14.914553 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:27:14.976 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:27:14.980165 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:27:25.074 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:27:25.077952 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:27:25.085 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:27:25.089059 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:27:35.193 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:27:35.196522 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:27:35.208 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:27:35.211948 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:27:45.325 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:27:45.328728 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:27:45.338 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:27:45.342090 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:27:55.469 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:27:55.471520 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:27:55.484 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:27:55.487334 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:28:05.604 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:28:05.606508 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:28:05.617 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:28:05.620528 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:28:15.696 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:28:15.698306 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:28:15.705 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:28:15.708782 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:28:25.820 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:28:25.822462 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:28:25.834 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:28:25.836672 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:28:35.951 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:28:35.953945 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:28:35.962 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:28:35.965459 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:28:46.064 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:28:46.066531 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:28:46.133 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:28:46.133270 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:28:56.213 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:28:56.215916 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:28:56.256 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:28:56.259400 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:29:06.364 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:29:06.365376 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:29:06.376 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:29:06.379209 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:29:16.485 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:29:16.487347 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:29:16.499 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:29:16.501761 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:29:26.585 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:29:26.587005 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:29:26.597 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:29:26.600129 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:29:36.707 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:29:36.708992 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:29:36.719 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:29:36.721640 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:29:46.834 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:29:46.836044 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:29:46.845 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:29:46.846994 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:29:56.933 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:29:56.935178 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:29:56.991 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:29:56.993337 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:30:07.116 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:30:07.118328 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:30:07.127 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:30:07.129353 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:30:17.218 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:30:17.218960 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:30:17.229 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:30:17.229679 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:30:27.299 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:30:27.299673 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:30:27.349 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:30:27.350819 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:30:37.430 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:30:37.431781 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:30:37.440 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:30:37.442178 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:30:47.518 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:30:47.519403 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:30:47.548 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:30:47.549469 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:30:57.659 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:30:57.660916 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:30:57.674 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:30:57.676269 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:31:07.807 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:31:07.808359 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:31:07.820 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:31:07.822126 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:31:17.929 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:31:17.930749 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:31:17.944 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:31:17.945739 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:31:28.106 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:31:28.107044 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:31:28.117 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:31:28.119104 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:31:38.190 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:31:38.191165 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:31:38.205 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:31:38.205794 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:31:48.376 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:31:48.376880 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:31:48.388 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:31:48.389952 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:31:58.501 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:31:58.501580 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:31:58.555 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:31:58.556308 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:32:08.639 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:32:08.639957 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:32:08.652 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:32:08.653452 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:32:18.766 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:32:18.766827 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:32:18.778 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:32:18.779538 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:32:28.885 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:32:28.882315 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:32:28.894 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:32:28.895222 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:32:39.004 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:32:39.005138 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:32:39.018 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:32:39.019599 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:32:49.136 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:32:49.136777 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:32:49.150 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:32:49.151246 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:32:59.228 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:32:59.228771 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:32:59.240 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:32:59.241081 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:33:09.386 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:33:09.387027 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:33:09.403 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:33:09.404097 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:33:19.500 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:33:19.501189 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:33:19.513 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:33:19.514657 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:33:29.604 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:33:29.605352 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:33:29.617 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:33:29.617742 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:33:39.690 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:33:39.690273 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:33:39.703 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:33:39.704476 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:33:49.812 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:33:49.812486 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:33:49.823 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:33:49.824281 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:33:59.919 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:33:59.920072 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:33:59.932 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:33:59.932742 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:34:10.000 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:34:10.001239 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:34:10.014 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:34:10.015072 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:34:20.090 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:34:20.090589 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:34:20.224 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:34:20.101803 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:34:30.185 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:34:30.185996 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:34:30.257 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:34:30.257880 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:34:40.326 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:34:40.326782 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:34:40.338 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:34:40.339496 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:34:50.510 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:34:50.511032 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:34:50.525 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:34:50.525740 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:35:00.625 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:35:00.625288 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:35:00.656 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:35:00.656950 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:35:10.722 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:35:10.721750 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:35:10.735 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:35:10.736159 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:35:20.802 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:35:20.802133 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:35:20.871 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:35:20.871830 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:35:30.966 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:35:30.966669 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:35:30.980 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:35:30.980353 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:35:41.102 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:35:41.102136 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:35:41.115 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:35:41.115441 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:35:51.202 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:35:51.201888 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:35:51.214 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:35:51.214774 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:36:01.289 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:36:01.289546 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:36:01.343 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:36:01.341545 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:36:11.445 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:36:11.444004 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:36:11.456 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:36:11.456453 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:36:21.525 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:36:21.525220 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:36:21.549 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:36:21.549671 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:36:31.679 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:36:31.675319 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:36:31.691 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:36:31.691631 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:36:41.764 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:36:41.764647 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:36:41.777 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:36:41.777342 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:36:51.848 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:36:51.847573 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:36:51.860 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:36:51.859582 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:37:01.982 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:37:01.982014 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:37:01.991 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:37:01.991396 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:37:12.056 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:37:12.056239 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:37:12.068 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:37:12.068405 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:37:22.189 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:37:22.188995 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:37:22.203 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:37:22.203240 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:37:32.266 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:37:32.266362 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:37:32.277 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:37:32.277079 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:37:42.353 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:37:42.353147 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:37:42.366 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:37:42.365747 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:37:52.441 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:37:52.440384 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:37:52.452 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:37:52.451620 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:38:02.519 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:38:02.519144 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:38:02.592 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:38:02.585814 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:38:12.667 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:38:12.666676 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:38:12.682 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:38:12.681656 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:38:22.747 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:38:22.746645 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:38:22.788 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:38:22.760698 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:38:32.850 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:38:32.850069 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:38:32.918 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:38:32.918092 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:38:42.993 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:38:42.992958 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:38:43.008 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:38:43.007435 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:38:53.086 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:38:53.085479 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:38:53.100 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:38:53.099865 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:39:03.196 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:39:03.195374 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:39:03.211 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:39:03.211267 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:39:13.354 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:39:13.353031 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:39:13.365 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:39:13.364627 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:39:23.446 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:39:23.445948 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:39:23.458 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:39:23.458037 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:39:33.527 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:39:33.522134 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:39:33.538 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:39:33.537329 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:39:43.599 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:39:43.597682 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:39:43.609 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:39:43.608427 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:39:53.672 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:39:53.671121 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:39:53.736 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:39:53.735121 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:40:03.814 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:40:03.813632 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:40:03.826 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:40:03.826237 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:40:13.947 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:40:13.946050 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:40:13.961 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:40:13.959586 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:40:24.029 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:40:24.027857 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:40:24.042 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:40:24.041598 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:40:34.111 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:40:34.110396 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:40:34.160 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:40:34.159266 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:40:44.286 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:40:44.285002 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:40:44.297 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:40:44.296909 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:40:54.397 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:40:54.396592 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:40:54.414 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:40:54.413112 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:41:04.474 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:41:04.473788 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:41:04.487 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:41:04.486355 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:41:14.589 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:41:14.588474 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:41:14.602 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:41:14.602008 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:41:24.705 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:41:24.703623 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:41:24.716 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:41:24.715795 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:41:34.780 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:41:34.779706 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:41:34.793 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:41:34.792605 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:41:44.889 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:41:44.887596 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:41:44.902 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:41:44.900509 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:41:55.016 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:41:55.014849 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:41:55.030 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:41:55.029450 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:42:05.138 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:42:05.137053 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:42:05.150 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:42:05.149002 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:42:05.247 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:42:05.245865 I | op-mon: parsing mon endpoints: a=10.100.186.100:6789,b=10.111.132.151:6789,c=10.101.46.100:6789 2022-03-10T06:42:05.961 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:42:05.959979 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T06:42:05.949+0000 7f88b47ce380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T06:42:05.962 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10T06:42:05.949+0000 7f88b47ce380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T06:42:05.962 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10T06:42:05.949+0000 7f88b47ce380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T06:42:05.962 INFO:tasks.rook.operator.smithi037.stdout:couldn't init storage provider. : exit status 5 2022-03-10T06:42:15.264 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:42:15.263017 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:42:15.279 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:42:15.278080 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:42:25.381 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:42:25.379366 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:42:25.396 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:42:25.395450 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:42:35.520 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:42:35.519202 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:42:35.533 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:42:35.531820 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:42:45.636 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:42:45.635261 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:42:45.651 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:42:45.650216 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:42:55.765 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:42:55.763376 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:42:55.780 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:42:55.778938 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:43:05.844 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:43:05.843212 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:43:05.888 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:43:05.886993 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:43:16.019 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:43:16.017454 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:43:16.031 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:43:16.030199 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:43:26.115 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:43:26.114313 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:43:26.126 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:43:26.125095 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:43:36.200 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:43:36.198725 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:43:36.215 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:43:36.214416 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:43:46.402 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:43:46.400552 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:43:46.415 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:43:46.413711 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:43:56.529 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:43:56.527542 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:43:56.546 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:43:56.545137 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:44:06.655 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:44:06.650376 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:44:06.708 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:44:06.707133 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:44:16.811 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:44:16.809429 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:44:16.823 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:44:16.822074 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:44:26.905 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:44:26.903275 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:44:26.919 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:44:26.917712 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:44:37.041 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:44:37.039125 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:44:37.052 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:44:37.050973 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:44:47.232 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:44:47.230385 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:44:47.246 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:44:47.244947 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:44:57.323 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:44:57.321332 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:44:57.338 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:44:57.337215 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:45:07.412 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:45:07.410737 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:45:07.456 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:45:07.454813 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:45:17.560 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:45:17.558451 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:45:17.574 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:45:17.573138 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:45:27.742 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:45:27.740228 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:45:27.752 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:45:27.750668 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:45:37.839 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:45:37.837352 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:45:37.852 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:45:37.851231 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:45:47.925 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:45:47.923290 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:45:47.938 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:45:47.936734 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:45:58.008 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:45:58.006378 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:45:58.020 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:45:58.018362 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:46:08.132 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:46:08.130472 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:46:08.144 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:46:08.141325 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:46:18.262 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:46:18.260112 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:46:18.273 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:46:18.271879 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:46:28.394 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:46:28.392528 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:46:28.409 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:46:28.407938 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:46:38.481 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:46:38.479732 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:46:38.495 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:46:38.493406 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:46:48.607 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:46:48.605637 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:46:48.622 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:46:48.618644 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:46:58.688 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:46:58.686599 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:46:58.701 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:46:58.699398 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:47:08.775 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:47:08.773713 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:47:08.790 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:47:08.788190 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:47:18.853 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:47:18.851289 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:47:18.862 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:47:18.860401 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:47:28.931 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:47:28.929195 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:47:28.943 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:47:28.941487 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:47:39.072 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:47:39.070138 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:47:39.229 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:47:39.084757 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:47:49.192 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:47:49.190758 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:47:49.205 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:47:49.203658 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:47:59.285 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:47:59.283269 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:47:59.336 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:47:59.334947 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:48:09.409 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:48:09.407611 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:48:09.423 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:48:09.421193 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:48:19.496 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:48:19.494170 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:48:19.509 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:48:19.507127 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:48:29.621 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:48:29.619648 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:48:29.634 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:48:29.632971 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:48:39.715 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:48:39.713753 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:48:39.766 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:48:39.764159 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:48:49.879 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:48:49.876860 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:48:49.891 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:48:49.889443 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:49:00.010 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:49:00.006671 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:49:00.024 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:49:00.021997 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:49:10.124 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:49:10.122784 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:49:10.141 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:49:10.139170 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:49:20.240 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:49:20.238479 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:49:20.254 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:49:20.252239 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:49:30.375 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:49:30.373246 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:49:30.387 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:49:30.385712 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:49:40.460 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:49:40.458348 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:49:40.474 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:49:40.472366 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:49:50.561 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:49:50.559143 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:49:50.575 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:49:50.573574 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:50:00.701 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:50:00.698569 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:50:00.712 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:50:00.710486 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:50:10.788 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:50:10.785765 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:50:10.802 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:50:10.800274 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:50:20.872 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:50:20.870663 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:50:20.885 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:50:20.883848 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:50:31.016 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:50:31.014121 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:50:31.026 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:50:31.024454 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:50:41.114 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:50:41.111869 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:50:41.137 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:50:41.135660 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:50:51.250 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:50:51.248116 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:50:51.259 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:50:51.257944 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:51:01.343 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:51:01.341218 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:51:01.356 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:51:01.353951 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:51:11.552 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:51:11.550034 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:51:11.561 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:51:11.559754 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:51:21.714 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:51:21.711402 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:51:21.726 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:51:21.724702 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:51:31.842 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:51:31.840111 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:51:31.854 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:51:31.852637 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:51:41.924 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:51:41.921882 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:51:41.971 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:51:41.969771 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:51:52.042 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:51:52.040093 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:51:52.093 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:51:52.091099 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:52:02.220 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:52:02.218041 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:52:02.232 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:52:02.230048 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:52:12.308 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:52:12.305851 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:52:12.320 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:52:12.317615 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:52:22.463 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:52:22.461089 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:52:22.476 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:52:22.473940 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:52:32.560 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:52:32.558401 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:52:32.572 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:52:32.570588 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:52:42.672 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:52:42.669835 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:52:42.683 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:52:42.681842 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:52:52.787 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:52:52.785128 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:52:52.800 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:52:52.798006 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:53:02.899 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:53:02.897278 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:53:02.918 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:53:02.916554 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:53:12.999 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:53:12.996745 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:53:13.049 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:53:13.047384 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:53:23.117 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:53:23.115418 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:53:23.129 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:53:23.127350 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:53:33.281 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:53:33.279444 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:53:33.295 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:53:33.293322 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:53:43.414 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:53:43.412060 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:53:43.427 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:53:43.425525 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:53:53.532 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:53:53.530606 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:53:53.548 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:53:53.543476 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:54:03.633 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:54:03.630541 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:54:03.646 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:54:03.644134 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:54:13.783 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:54:13.781196 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:54:13.797 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:54:13.795495 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:54:23.969 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:54:23.967383 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:54:23.982 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:54:23.980453 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:54:34.071 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:54:34.069076 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:54:34.084 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:54:34.082081 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:54:44.183 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:54:44.180445 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:54:44.196 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:54:44.194750 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:54:54.307 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:54:54.305302 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:54:54.320 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:54:54.318317 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:55:04.378 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:55:04.375186 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:55:04.393 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:55:04.390865 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:55:14.548 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:55:14.546043 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:55:14.561 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:55:14.558670 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:55:24.660 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:55:24.657409 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:55:24.671 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:55:24.669471 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:55:34.745 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:55:34.743091 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:55:34.758 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:55:34.756626 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:55:44.865 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:55:44.863202 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:55:44.881 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:55:44.878641 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:55:54.956 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:55:54.953695 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:55:54.971 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:55:54.969082 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:56:05.044 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:56:05.041756 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:56:05.057 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:56:05.055076 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:56:15.175 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:56:15.173072 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:56:15.186 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:56:15.184340 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:56:25.263 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:56:25.260751 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:56:25.277 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:56:25.274807 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:56:35.351 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:56:35.349062 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:56:35.364 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:56:35.362041 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:56:45.438 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:56:45.435864 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:56:45.452 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:56:45.450027 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:56:55.543 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:56:55.540791 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:56:55.556 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:56:55.554308 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:57:05.643 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:57:05.640301 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:57:05.690 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:57:05.687750 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:57:15.761 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:57:15.759092 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:57:15.772 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:57:15.769862 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:57:25.895 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:57:25.893422 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:57:25.912 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:57:25.910299 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:57:36.033 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:57:36.031947 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:57:36.048 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:57:36.045682 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:57:46.117 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:57:46.116386 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:57:46.130 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:57:46.128885 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:57:56.211 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:57:56.209961 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:57:56.224 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:57:56.223114 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:58:06.435 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:58:06.433747 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:58:06.446 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:58:06.445773 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:58:16.542 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:58:16.542099 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:58:16.553 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:58:16.552862 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:58:26.727 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:58:26.727492 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:58:26.741 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:58:26.740655 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:58:36.936 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:58:36.935392 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:58:36.947 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:58:36.947847 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:58:45.965 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:58:45.965321 I | op-mon: parsing mon endpoints: a=10.100.186.100:6789,b=10.111.132.151:6789,c=10.101.46.100:6789 2022-03-10T06:58:46.687 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:58:46.687983 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T06:58:46.677+0000 7ff875b79380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T06:58:46.688 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10T06:58:46.677+0000 7ff875b79380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T06:58:46.688 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10T06:58:46.677+0000 7ff875b79380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T06:58:46.688 INFO:tasks.rook.operator.smithi037.stdout:couldn't init storage provider. : exit status 5 2022-03-10T06:58:47.033 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:58:47.033554 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:58:47.043 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:58:47.044492 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:58:57.183 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:58:57.183988 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:58:57.196 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:58:57.197614 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:59:07.309 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:59:07.310139 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:59:07.323 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:59:07.325227 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:59:17.400 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:59:17.401758 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:59:17.412 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:59:17.414238 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:59:27.483 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:59:27.485012 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:59:27.495 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:59:27.497684 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:59:37.575 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:59:37.577649 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:59:37.590 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:59:37.593087 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:59:47.675 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:59:47.677826 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:59:47.688 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:59:47.691467 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:59:57.825 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:59:57.828524 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T06:59:57.842 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 06:59:57.845300 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T07:00:07.913 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 07:00:07.916405 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T07:00:07.958 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 07:00:07.961078 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T07:00:18.063 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 07:00:18.066638 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T07:00:18.076 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 07:00:18.079798 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T07:00:28.181 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 07:00:28.184580 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T07:00:28.196 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 07:00:28.200679 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T07:00:38.265 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 07:00:38.269677 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T07:00:38.281 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 07:00:38.284389 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T07:00:48.347 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 07:00:48.351779 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T07:00:48.416 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 07:00:48.420128 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T07:00:58.482 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 07:00:58.486401 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T07:00:58.495 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 07:00:58.499446 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T07:01:08.597 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 07:01:08.601893 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T07:01:08.609 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 07:01:08.614959 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T07:01:18.680 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 07:01:18.685546 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T07:01:18.701 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 07:01:18.699836 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T07:01:28.790 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 07:01:28.795924 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T07:01:28.804 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 07:01:28.809706 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T07:01:38.880 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 07:01:38.885649 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T07:01:38.894 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 07:01:38.899794 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T07:01:49.002 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 07:01:49.008229 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T07:01:49.016 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 07:01:49.022360 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T07:01:59.115 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 07:01:59.121122 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T07:01:59.126 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 07:01:59.132729 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T07:02:09.196 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 07:02:09.202486 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T07:02:09.211 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 07:02:09.217904 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T07:02:19.349 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 07:02:19.356296 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T07:02:19.364 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 07:02:19.371728 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T07:02:29.489 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 07:02:29.495941 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T07:02:29.500 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 07:02:29.507342 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T07:02:39.567 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 07:02:39.574817 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T07:02:39.578 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 07:02:39.585762 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T07:02:49.650 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 07:02:49.658208 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T07:02:49.659 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 07:02:49.667013 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T07:02:59.796 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 07:02:59.803880 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T07:02:59.809 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 07:02:59.816444 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T07:03:09.874 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 07:03:09.881880 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T07:03:09.887 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 07:03:09.895574 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T07:03:19.990 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 07:03:19.998227 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T07:03:20.004 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 07:03:20.011968 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T07:03:30.115 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 07:03:30.123329 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T07:03:30.127 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 07:03:30.135864 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T07:03:40.193 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 07:03:40.201531 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T07:03:40.206 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 07:03:40.214987 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T07:03:50.303 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 07:03:50.312331 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T07:03:50.316 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 07:03:50.324861 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T07:04:00.427 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 07:04:00.436136 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T07:04:00.439 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 07:04:00.447945 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T07:04:10.551 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 07:04:10.559808 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T07:04:10.567 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 07:04:10.574972 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T07:04:20.709 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 07:04:20.718812 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T07:04:20.723 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 07:04:20.733217 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T07:04:30.891 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 07:04:30.899634 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T07:04:30.904 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 07:04:30.913734 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T07:04:41.043 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 07:04:41.053103 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T07:04:41.053 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 07:04:41.063350 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T07:04:51.115 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 07:04:51.124800 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T07:04:51.154 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 07:04:51.164295 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T07:05:01.222 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 07:05:01.231410 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T07:05:01.232 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 07:05:01.243040 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T07:05:11.326 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 07:05:11.335830 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T07:05:11.336 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 07:05:11.346975 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T07:05:21.475 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 07:05:21.483698 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T07:05:21.486 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 07:05:21.497032 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T07:05:31.545 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 07:05:31.551804 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T07:05:31.552 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 07:05:31.563227 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T07:05:41.653 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 07:05:41.663990 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T07:05:41.664 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 07:05:41.675269 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T07:05:51.780 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 07:05:51.791280 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T07:05:51.795 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 07:05:51.805303 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T07:06:01.891 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 07:06:01.902822 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T07:06:01.909 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 07:06:01.921079 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T07:06:11.976 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 07:06:11.987527 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T07:06:11.987 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 07:06:11.999251 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T07:06:22.053 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 07:06:22.064858 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T07:06:22.065 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 07:06:22.076451 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T07:06:32.142 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 07:06:32.153712 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T07:06:32.153 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 07:06:32.164874 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T07:06:42.216 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 07:06:42.225230 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T07:06:42.227 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 07:06:42.239452 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T07:06:52.353 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 07:06:52.364854 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T07:06:52.364 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 07:06:52.376441 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T07:07:02.462 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 07:07:02.474483 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T07:07:02.475 INFO:tasks.rook.operator.smithi037.stdout:2022-03-10 07:07:02.487133 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T09:28:10.527 DEBUG:teuthology.exit:Got signal 15; running 1 handler... 2022-03-10T09:28:10.558 DEBUG:teuthology.task.console_log:Killing console logger for smithi037 2022-03-10T09:28:10.559 DEBUG:teuthology.task.console_log:Killing console logger for smithi038 2022-03-10T09:28:10.561 DEBUG:teuthology.task.console_log:Killing console logger for smithi195 2022-03-10T09:28:10.561 DEBUG:teuthology.exit:Finished running handlers