2022-03-08T22:51:59.115 INFO:root:teuthology version: 1.1.0-eeaeee38 2022-03-08T22:51:59.117 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2022-03-08_22:38:39-rados-wip-yuri5-testing-2022-03-07-0958-distro-default-smithi/6727295 --description rados/rook/smoke/{0-distro/ubuntu_20.04 0-kubeadm 0-nvme-loop 1-rook 2-workload/none 3-final cluster/3-node k8s/1.21 net/flannel rook/master} --verbose --name yuriw-2022-03-08_22:38:39-rados-wip-yuri5-testing-2022-03-07-0958-distro-default-smithi -- /home/teuthworker/archive/yuriw-2022-03-08_22:38:39-rados-wip-yuri5-testing-2022-03-07-0958-distro-default-smithi/6727295/orig.config.yaml 2022-03-08T22:51:59.153 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2022-03-08T22:51:59.216 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2022-03-08_22:38:39-rados-wip-yuri5-testing-2022-03-07-0958-distro-default-smithi/6727295 branch: wip-yuri5-testing-2022-03-07-0958 description: rados/rook/smoke/{0-distro/ubuntu_20.04 0-kubeadm 0-nvme-loop 1-rook 2-workload/none 3-final cluster/3-node k8s/1.21 net/flannel rook/master} email: null first_in_suite: false job_id: '6727295' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: yuriw-2022-03-08_22:38:39-rados-wip-yuri5-testing-2022-03-07-0958-distro-default-smithi nuke-on-error: true os_type: ubuntu os_version: '20.04' overrides: admin_socket: branch: wip-yuri5-testing-2022-03-07-0958 ceph: conf: mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 7a44e9ebca556862e9704be295a07719d7db47b3 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: flavor: default sha1: 7a44e9ebca556862e9704be295a07719d7db47b3 kernel: hwe: true kubeadm: pod_network: flannel version: '1.21' rook: rook_image: rook/ceph:master workunit: branch: wip-yuri5-testing-2022-03-07-0958 sha1: 7a44e9ebca556862e9704be295a07719d7db47b3 owner: scheduled_yuriw@teuthology priority: 71 repo: https://github.com/ceph/ceph-ci.git roles: - - host.a - client.a - - host.b - client.b - - host.c - client.c sha1: 7a44e9ebca556862e9704be295a07719d7db47b3 sleep_before_teardown: 0 suite: rados suite_branch: wip-yuri5-testing-2022-03-07-0958 suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_7a44e9ebca556862e9704be295a07719d7db47b3/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 7a44e9ebca556862e9704be295a07719d7db47b3 targets: smithi077.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDOdWPCncdjLuuHtZCApGQk1ePG9b5+5+OSkG9iBLWd+YaZhFppthcChhBtRFZSEPVLS00BeLcjh5HM4/2+f7qPzJkaTORFvVRkx/EwiONXJT9SiBL5BnXICKutvgML9TiYhwuWboZDMxiLPj1wFOP+nRRR4hpH4fIoodN+wIbTaEbTQ1NjdYMgmm2VPKYt2VPlGFhDsOm1K+TyrVrcZ0AAFXux/xObfiFBapbzRbkrztsh5MotwomEUXaY98jnlZ/2pihjgwd6pSbvAHSKyXLhJLpumGDUGhuUr4tjnVAZFbsbMPbFIuMP+Kop6mPUbEg1GV0818Aaqy/fXrvIjMcMxYYclGmpTFbutOf82MDsnOf1+2W9Ak8cZWRaJkE4NnfW9Ysk8MAYovpIHYhG0ovpIz6f7ZhLac7nVwc0QDHQLkLzL0RhHHHPKJPG9w9mQ86nJudQRRxELPrsyZ52h7bSNPCuPLW88C42bgjT9cWYKiw8iJZnDkBAty/k/SE8dZE= smithi106.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCv/TFMqIgrtUtVltvdUWV+VWSwq4YWkqbkEwTh5Oe5HCFjbyGG/2Qxocu6E7kYwQ5J4YWglM1REemHMjFE3ipSXumEC1zNQiaBIK5rH+6SN1Ff+wkXhSDCmDxqwVc0ryljHeTMfDyF1hmoJcfx07RX22ROFazC7EuK6Q3LrQoei81sD7oygqS8IoH/WWyQ1YHqWxX+W6GXTWHc9wYximAlKtratjyJaqAW6mEG2YoYBhynEYSg39exSiIQ+W7Fy93DW1Wu/pDqVV5gvt/xsi8uoagZd8yASAwVx3pM/ihjQqBO1AqvMZI8tUQOKZ2gxWs5xyvp56oy8SfU5f1qt+z3M+UKCa9uH67t/dXi7ObY95VUOZjCmLFDgQj9/3IIe4zL7WADieCdQAGYnWsP73G1/OSNkHE7mR8pEMN15Ug0yVaVDytfW48kArCtWHZcgfLGDRoIh1ssF96hiCedHNy3B8ys4pf7vWKjJ8D7yqYj31kJ+SyYw+TWir+1MBdtUt8= smithi135.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC/cBKS2oePijLDv9QMHREtRi0Jm5PwQy6oS6uW9NyoWD1tJbdRAuZVpMy0K2scOGAAjGllCS7rlTPl7x+As0HJz19iLugokY+pSWP4NQW0uh41Uf2lhgqSYREv88UZxxeW2F+vVVby38cUSvugrCs73liirXEu6OX7m3+O9dD6pc7KEcHfAyAYHJpItTI9EwV5aeVxKpV7+9x7Qg2V9xBoViinnHLvnwd32tcwJgiOIxPrQ9mUJqdQet6D+0pKTpi26Ps/beW5NB0Io3T+iAjQ28s217wa3gJPsIsiMxICig/wQeFcBF9KL1SY+5U99ksXpwhzR6wrkFkSCe3z1z/pqeqH0RVu1bR2XVRVpKuxfjV/tzb89c9wx+rhvjM6gF3Bc9yRE099d6jtcJU3kLWES1+wJxAY3YqhyE0p9pCobpXm8ABBivi7m54FnmE29aJ8yoI9b30XO9wJOiJW7ka71fWKmgV0hpfiLtG2/hVNa0lDOrjtsEU0aWjH7rMA+Ds= tasks: - kubeadm: null - nvme_loop: null - rook: null - rook.shell: - ceph -s - ceph orch status - ceph orch ps - ceph orch ls - ceph orch device ls - exec: host.a: - "set -ex\ntoolbox() {\n kubectl -n rook-ceph exec -it deploy/rook-ceph-tools\ \ -- \"$@\"\n}\norig_num_osd=`toolbox ceph osd stat | cut -f3 -d \" \"`\ntoolbox\ \ ceph orch osd rm 0 --force\nremoved_pv=\"\"\nwhile [ \"$removed_pv\" = \"\"\ \ ]\ndo\n removed_pv=`kubectl get pv | grep Released | cut -f1 -d \" \"`\n\ \ sleep 3s\ndone\ntarget_path=`kubectl get pv $removed_pv -o jsonpath='{.spec.local.path}'`\n\ host=`echo $removed_pv | cut -f1 -d \"-\"`\ntoolbox ceph orch device zap $host\ \ $target_path --force\nzap_completion=\"0\"\nwhile [ \"$zap_completion\" =\ \ \"0\" ]\ndo\n zap_completion=`kubectl get job -n rook-ceph rook-ceph-device-zap\ \ -o jsonpath='{.status.succeeded.path}'`\n sleep 3s\ndone\nkubectl patch\ \ pv $removed_pv -p '{\"spec\":{\"claimRef\": null}}'\ntoolbox ceph orch apply\ \ osd --all-available-devices\nkubectl delete job rook-ceph-device-zap -n rook-ceph\n\ num_osd=\"0\"\nwhile [ \"$num_osd\" != \"$orig_num_osd\" ]\ndo\n echo \"\ waiting for osd to come back up\"\n num_osd=`toolbox ceph osd stat | cut\ \ -f3 -d \" \"`\n sleep 30s\ndone\n" - rook.shell: commands: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - ceph orch apply rgw foo - ceph orch apply mds foo - ceph orch apply rbd-mirror - ceph orch apply nfs foo --port 12777 teuthology_branch: master teuthology_sha1: eeaeee38264fad8a01fd1ce912d8908676ed64bc timestamp: 2022-03-08_22:38:39 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.22301 2022-03-08T22:51:59.217 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_7a44e9ebca556862e9704be295a07719d7db47b3/qa; will attempt to use it 2022-03-08T22:51:59.217 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_7a44e9ebca556862e9704be295a07719d7db47b3/qa/tasks 2022-03-08T22:51:59.218 INFO:teuthology.run_tasks:Running task internal.check_packages... 2022-03-08T22:51:59.220 INFO:teuthology.task.internal:Checking packages... 2022-03-08T22:51:59.241 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '7a44e9ebca556862e9704be295a07719d7db47b3' 2022-03-08T22:51:59.241 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2022-03-08T22:51:59.242 INFO:teuthology.packaging:ref: None 2022-03-08T22:51:59.242 INFO:teuthology.packaging:tag: None 2022-03-08T22:51:59.242 INFO:teuthology.packaging:branch: wip-yuri5-testing-2022-03-07-0958 2022-03-08T22:51:59.243 INFO:teuthology.packaging:sha1: 7a44e9ebca556862e9704be295a07719d7db47b3 2022-03-08T22:51:59.243 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&ref=wip-yuri5-testing-2022-03-07-0958 2022-03-08T22:51:59.417 INFO:teuthology.task.internal:Found packages for ceph version 17.0.0-11025-g7a44e9eb-1focal 2022-03-08T22:51:59.417 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2022-03-08T22:51:59.428 INFO:teuthology.task.internal:no buildpackages task found 2022-03-08T22:51:59.428 INFO:teuthology.run_tasks:Running task internal.save_config... 2022-03-08T22:51:59.480 INFO:teuthology.task.internal:Saving configuration 2022-03-08T22:51:59.493 INFO:teuthology.run_tasks:Running task internal.check_lock... 2022-03-08T22:51:59.507 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-03-08T22:51:59.531 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi077.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2022-03-08_22:38:39-rados-wip-yuri5-testing-2022-03-07-0958-distro-default-smithi/6727295', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2022-03-08 22:45:50.400786', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIAZz+wNmqb0lT7QJrX5pHqlsT+ctai9+1cqkwrFawle0'} 2022-03-08T22:51:59.556 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi106.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2022-03-08_22:38:39-rados-wip-yuri5-testing-2022-03-07-0958-distro-default-smithi/6727295', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2022-03-08 22:45:50.403324', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIFypM2M9vadrfETAxj6S+Z3aNWfjsqYszpUbDWXTH28Y'} 2022-03-08T22:51:59.579 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi135.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2022-03-08_22:38:39-rados-wip-yuri5-testing-2022-03-07-0958-distro-default-smithi/6727295', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2022-03-08 22:45:50.402181', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIJ2OhELJGlQOdgCxgkgXTiBlBMmTgCyzd8mE2enRNYrN'} 2022-03-08T22:51:59.579 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2022-03-08T22:51:59.590 INFO:teuthology.task.internal:roles: ubuntu@smithi077.front.sepia.ceph.com - ['host.a', 'client.a'] 2022-03-08T22:51:59.590 INFO:teuthology.task.internal:roles: ubuntu@smithi106.front.sepia.ceph.com - ['host.b', 'client.b'] 2022-03-08T22:51:59.590 INFO:teuthology.task.internal:roles: ubuntu@smithi135.front.sepia.ceph.com - ['host.c', 'client.c'] 2022-03-08T22:51:59.591 INFO:teuthology.run_tasks:Running task console_log... 2022-03-08T22:51:59.718 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fa04b36e9d8>, signals=[15]) 2022-03-08T22:51:59.720 INFO:teuthology.run_tasks:Running task internal.connect... 2022-03-08T22:51:59.730 INFO:teuthology.task.internal:Opening connections... 2022-03-08T22:51:59.731 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi077.front.sepia.ceph.com 2022-03-08T22:51:59.733 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-08T22:51:59.784 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi106.front.sepia.ceph.com 2022-03-08T22:51:59.785 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi106.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-08T22:51:59.836 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi135.front.sepia.ceph.com 2022-03-08T22:51:59.837 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi135.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-08T22:51:59.885 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2022-03-08T22:51:59.894 DEBUG:teuthology.orchestra.run.smithi077:> uname -m 2022-03-08T22:51:59.901 INFO:teuthology.orchestra.run.smithi077.stdout:x86_64 2022-03-08T22:51:59.902 DEBUG:teuthology.orchestra.run.smithi077:> cat /etc/os-release 2022-03-08T22:51:59.951 INFO:teuthology.orchestra.run.smithi077.stdout:NAME="Ubuntu" 2022-03-08T22:51:59.951 INFO:teuthology.orchestra.run.smithi077.stdout:VERSION="20.04.3 LTS (Focal Fossa)" 2022-03-08T22:51:59.952 INFO:teuthology.orchestra.run.smithi077.stdout:ID=ubuntu 2022-03-08T22:51:59.952 INFO:teuthology.orchestra.run.smithi077.stdout:ID_LIKE=debian 2022-03-08T22:51:59.952 INFO:teuthology.orchestra.run.smithi077.stdout:PRETTY_NAME="Ubuntu 20.04.3 LTS" 2022-03-08T22:51:59.952 INFO:teuthology.orchestra.run.smithi077.stdout:VERSION_ID="20.04" 2022-03-08T22:51:59.953 INFO:teuthology.orchestra.run.smithi077.stdout:HOME_URL="https://www.ubuntu.com/" 2022-03-08T22:51:59.953 INFO:teuthology.orchestra.run.smithi077.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2022-03-08T22:51:59.953 INFO:teuthology.orchestra.run.smithi077.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2022-03-08T22:51:59.953 INFO:teuthology.orchestra.run.smithi077.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2022-03-08T22:51:59.953 INFO:teuthology.orchestra.run.smithi077.stdout:VERSION_CODENAME=focal 2022-03-08T22:51:59.954 INFO:teuthology.orchestra.run.smithi077.stdout:UBUNTU_CODENAME=focal 2022-03-08T22:51:59.954 INFO:teuthology.lock.ops:Updating smithi077.front.sepia.ceph.com on lock server 2022-03-08T22:51:59.974 DEBUG:teuthology.orchestra.run.smithi106:> uname -m 2022-03-08T22:51:59.981 INFO:teuthology.orchestra.run.smithi106.stdout:x86_64 2022-03-08T22:51:59.981 DEBUG:teuthology.orchestra.run.smithi106:> cat /etc/os-release 2022-03-08T22:51:59.988 INFO:teuthology.orchestra.run.smithi106.stdout:NAME="Ubuntu" 2022-03-08T22:51:59.988 INFO:teuthology.orchestra.run.smithi106.stdout:VERSION="20.04.3 LTS (Focal Fossa)" 2022-03-08T22:51:59.988 INFO:teuthology.orchestra.run.smithi106.stdout:ID=ubuntu 2022-03-08T22:51:59.989 INFO:teuthology.orchestra.run.smithi106.stdout:ID_LIKE=debian 2022-03-08T22:51:59.989 INFO:teuthology.orchestra.run.smithi106.stdout:PRETTY_NAME="Ubuntu 20.04.3 LTS" 2022-03-08T22:51:59.989 INFO:teuthology.orchestra.run.smithi106.stdout:VERSION_ID="20.04" 2022-03-08T22:51:59.990 INFO:teuthology.orchestra.run.smithi106.stdout:HOME_URL="https://www.ubuntu.com/" 2022-03-08T22:51:59.990 INFO:teuthology.orchestra.run.smithi106.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2022-03-08T22:51:59.991 INFO:teuthology.orchestra.run.smithi106.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2022-03-08T22:51:59.991 INFO:teuthology.orchestra.run.smithi106.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2022-03-08T22:51:59.991 INFO:teuthology.orchestra.run.smithi106.stdout:VERSION_CODENAME=focal 2022-03-08T22:51:59.991 INFO:teuthology.orchestra.run.smithi106.stdout:UBUNTU_CODENAME=focal 2022-03-08T22:51:59.992 INFO:teuthology.lock.ops:Updating smithi106.front.sepia.ceph.com on lock server 2022-03-08T22:52:00.008 DEBUG:teuthology.orchestra.run.smithi135:> uname -m 2022-03-08T22:52:00.015 INFO:teuthology.orchestra.run.smithi135.stdout:x86_64 2022-03-08T22:52:00.016 DEBUG:teuthology.orchestra.run.smithi135:> cat /etc/os-release 2022-03-08T22:52:00.022 INFO:teuthology.orchestra.run.smithi135.stdout:NAME="Ubuntu" 2022-03-08T22:52:00.023 INFO:teuthology.orchestra.run.smithi135.stdout:VERSION="20.04.3 LTS (Focal Fossa)" 2022-03-08T22:52:00.023 INFO:teuthology.orchestra.run.smithi135.stdout:ID=ubuntu 2022-03-08T22:52:00.023 INFO:teuthology.orchestra.run.smithi135.stdout:ID_LIKE=debian 2022-03-08T22:52:00.024 INFO:teuthology.orchestra.run.smithi135.stdout:PRETTY_NAME="Ubuntu 20.04.3 LTS" 2022-03-08T22:52:00.024 INFO:teuthology.orchestra.run.smithi135.stdout:VERSION_ID="20.04" 2022-03-08T22:52:00.024 INFO:teuthology.orchestra.run.smithi135.stdout:HOME_URL="https://www.ubuntu.com/" 2022-03-08T22:52:00.025 INFO:teuthology.orchestra.run.smithi135.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2022-03-08T22:52:00.025 INFO:teuthology.orchestra.run.smithi135.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2022-03-08T22:52:00.026 INFO:teuthology.orchestra.run.smithi135.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2022-03-08T22:52:00.026 INFO:teuthology.orchestra.run.smithi135.stdout:VERSION_CODENAME=focal 2022-03-08T22:52:00.026 INFO:teuthology.orchestra.run.smithi135.stdout:UBUNTU_CODENAME=focal 2022-03-08T22:52:00.027 INFO:teuthology.lock.ops:Updating smithi135.front.sepia.ceph.com on lock server 2022-03-08T22:52:00.044 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2022-03-08T22:52:00.060 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2022-03-08T22:52:00.082 INFO:teuthology.task.internal:Checking for old test directory... 2022-03-08T22:52:00.083 DEBUG:teuthology.orchestra.run.smithi077:> test '!' -e /home/ubuntu/cephtest 2022-03-08T22:52:00.086 DEBUG:teuthology.orchestra.run.smithi106:> test '!' -e /home/ubuntu/cephtest 2022-03-08T22:52:00.088 DEBUG:teuthology.orchestra.run.smithi135:> test '!' -e /home/ubuntu/cephtest 2022-03-08T22:52:00.094 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2022-03-08T22:52:00.108 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2022-03-08T22:52:00.108 DEBUG:teuthology.orchestra.run.smithi077:> test -z $(ls -A /var/lib/ceph) 2022-03-08T22:52:00.135 DEBUG:teuthology.orchestra.run.smithi106:> test -z $(ls -A /var/lib/ceph) 2022-03-08T22:52:00.137 DEBUG:teuthology.orchestra.run.smithi135:> test -z $(ls -A /var/lib/ceph) 2022-03-08T22:52:00.152 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2022-03-08T22:52:00.229 INFO:teuthology.run_tasks:Running task kernel... 2022-03-08T22:52:00.246 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2022-03-08T22:52:00.247 INFO:teuthology.task.kernel:normalize config orig: {'hwe': True} 2022-03-08T22:52:00.247 DEBUG:teuthology.task.kernel:normalized overrides {'host.a': {'hwe': True}, 'client.a': {'hwe': True}, 'host.b': {'hwe': True}, 'client.b': {'hwe': True}, 'host.c': {'hwe': True}, 'client.c': {'hwe': True}} 2022-03-08T22:52:00.248 INFO:teuthology.task.kernel:config {'host.a': {'kdb': True, 'sha1': 'distro', 'hwe': True}, 'host.b': {'kdb': True, 'sha1': 'distro', 'hwe': True}, 'host.c': {'kdb': True, 'sha1': 'distro', 'hwe': True}}, timeout 300 2022-03-08T22:52:00.248 DEBUG:teuthology.orchestra.run.smithi077:> uname -r 2022-03-08T22:52:00.249 DEBUG:teuthology.orchestra.run.smithi106:> uname -r 2022-03-08T22:52:00.250 DEBUG:teuthology.orchestra.run.smithi135:> uname -r 2022-03-08T22:52:00.256 INFO:teuthology.orchestra.run.smithi106.stdout:5.4.0-97-generic 2022-03-08T22:52:00.257 INFO:teuthology.task.kernel:Running kernel on smithi106: 5.4.0-97-generic 2022-03-08T22:52:00.257 DEBUG:teuthology.orchestra.run.smithi106:> sudo apt-get clean 2022-03-08T22:52:00.259 INFO:teuthology.orchestra.run.smithi135.stdout:5.4.0-97-generic 2022-03-08T22:52:00.259 INFO:teuthology.orchestra.run.smithi077.stdout:5.4.0-97-generic 2022-03-08T22:52:00.260 INFO:teuthology.task.kernel:Running kernel on smithi135: 5.4.0-97-generic 2022-03-08T22:52:00.260 DEBUG:teuthology.orchestra.run.smithi135:> sudo apt-get clean 2022-03-08T22:52:00.261 INFO:teuthology.task.kernel:Running kernel on smithi077: 5.4.0-97-generic 2022-03-08T22:52:00.261 DEBUG:teuthology.orchestra.run.smithi077:> sudo apt-get clean 2022-03-08T22:52:00.370 DEBUG:teuthology.orchestra.run.smithi077:> sudo apt-get update 2022-03-08T22:52:00.390 DEBUG:teuthology.orchestra.run.smithi106:> sudo apt-get update 2022-03-08T22:52:00.426 DEBUG:teuthology.orchestra.run.smithi135:> sudo apt-get update 2022-03-08T22:52:00.655 INFO:teuthology.orchestra.run.smithi077.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-08T22:52:00.655 INFO:teuthology.orchestra.run.smithi077.stdout:Get:2 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2022-03-08T22:52:00.684 INFO:teuthology.orchestra.run.smithi106.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-08T22:52:00.687 INFO:teuthology.orchestra.run.smithi106.stdout:Get:2 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2022-03-08T22:52:00.718 INFO:teuthology.orchestra.run.smithi077.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2022-03-08T22:52:00.734 INFO:teuthology.orchestra.run.smithi135.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-08T22:52:00.734 INFO:teuthology.orchestra.run.smithi135.stdout:Get:2 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2022-03-08T22:52:00.738 INFO:teuthology.orchestra.run.smithi106.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2022-03-08T22:52:00.791 INFO:teuthology.orchestra.run.smithi135.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2022-03-08T22:52:01.058 INFO:teuthology.orchestra.run.smithi077.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2022-03-08T22:52:01.076 INFO:teuthology.orchestra.run.smithi106.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2022-03-08T22:52:01.126 INFO:teuthology.orchestra.run.smithi135.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2022-03-08T22:52:01.135 INFO:teuthology.orchestra.run.smithi106.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [395 kB] 2022-03-08T22:52:01.142 INFO:teuthology.orchestra.run.smithi077.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [1,310 kB] 2022-03-08T22:52:01.211 INFO:teuthology.orchestra.run.smithi135.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [1,310 kB] 2022-03-08T22:52:01.287 INFO:teuthology.orchestra.run.smithi077.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [1,640 kB] 2022-03-08T22:52:01.288 INFO:teuthology.orchestra.run.smithi106.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [615 kB] 2022-03-08T22:52:01.351 INFO:teuthology.orchestra.run.smithi135.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [615 kB] 2022-03-08T22:52:01.452 INFO:teuthology.orchestra.run.smithi106.stdout:Get:7 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [1,310 kB] 2022-03-08T22:52:01.565 INFO:teuthology.orchestra.run.smithi106.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [1,640 kB] 2022-03-08T22:52:01.625 INFO:teuthology.orchestra.run.smithi077.stdout:Get:7 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [395 kB] 2022-03-08T22:52:01.646 INFO:teuthology.orchestra.run.smithi077.stdout:Get:8 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [228 kB] 2022-03-08T22:52:01.652 INFO:teuthology.orchestra.run.smithi135.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [1,640 kB] 2022-03-08T22:52:01.652 INFO:teuthology.orchestra.run.smithi135.stdout:Get:8 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [395 kB] 2022-03-08T22:52:01.656 INFO:teuthology.orchestra.run.smithi077.stdout:Get:9 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [9,732 B] 2022-03-08T22:52:01.658 INFO:teuthology.orchestra.run.smithi077.stdout:Get:10 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [799 kB] 2022-03-08T22:52:01.662 INFO:teuthology.orchestra.run.smithi106.stdout:Get:9 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [228 kB] 2022-03-08T22:52:01.681 INFO:teuthology.orchestra.run.smithi135.stdout:Get:9 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [228 kB] 2022-03-08T22:52:01.685 INFO:teuthology.orchestra.run.smithi077.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [615 kB] 2022-03-08T22:52:01.686 INFO:teuthology.orchestra.run.smithi106.stdout:Get:10 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [9,732 B] 2022-03-08T22:52:01.686 INFO:teuthology.orchestra.run.smithi106.stdout:Get:11 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [799 kB] 2022-03-08T22:52:01.694 INFO:teuthology.orchestra.run.smithi135.stdout:Get:10 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [9,732 B] 2022-03-08T22:52:01.695 INFO:teuthology.orchestra.run.smithi135.stdout:Get:11 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [799 kB] 2022-03-08T22:52:01.838 INFO:teuthology.orchestra.run.smithi077.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [312 kB] 2022-03-08T22:52:01.840 INFO:teuthology.orchestra.run.smithi135.stdout:Get:12 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [21.7 kB] 2022-03-08T22:52:01.840 INFO:teuthology.orchestra.run.smithi135.stdout:Get:13 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [114 kB] 2022-03-08T22:52:01.841 INFO:teuthology.orchestra.run.smithi077.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [14.8 kB] 2022-03-08T22:52:01.842 INFO:teuthology.orchestra.run.smithi077.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [853 kB] 2022-03-08T22:52:01.843 INFO:teuthology.orchestra.run.smithi135.stdout:Get:14 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [504 B] 2022-03-08T22:52:01.843 INFO:teuthology.orchestra.run.smithi135.stdout:Get:15 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [681 kB] 2022-03-08T22:52:01.862 INFO:teuthology.orchestra.run.smithi077.stdout:Get:15 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [21.7 kB] 2022-03-08T22:52:01.863 INFO:teuthology.orchestra.run.smithi077.stdout:Get:16 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [114 kB] 2022-03-08T22:52:01.866 INFO:teuthology.orchestra.run.smithi106.stdout:Get:12 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [21.7 kB] 2022-03-08T22:52:01.867 INFO:teuthology.orchestra.run.smithi077.stdout:Get:17 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [504 B] 2022-03-08T22:52:01.867 INFO:teuthology.orchestra.run.smithi077.stdout:Get:18 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [536 kB] 2022-03-08T22:52:01.867 INFO:teuthology.orchestra.run.smithi106.stdout:Get:13 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [114 kB] 2022-03-08T22:52:01.869 INFO:teuthology.orchestra.run.smithi106.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [312 kB] 2022-03-08T22:52:01.870 INFO:teuthology.orchestra.run.smithi106.stdout:Get:15 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [504 B] 2022-03-08T22:52:01.870 INFO:teuthology.orchestra.run.smithi106.stdout:Get:16 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [681 kB] 2022-03-08T22:52:01.880 INFO:teuthology.orchestra.run.smithi077.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [23.1 kB] 2022-03-08T22:52:01.880 INFO:teuthology.orchestra.run.smithi077.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [122 kB] 2022-03-08T22:52:01.883 INFO:teuthology.orchestra.run.smithi077.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [500 B] 2022-03-08T22:52:01.883 INFO:teuthology.orchestra.run.smithi077.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [672 kB] 2022-03-08T22:52:01.883 INFO:teuthology.orchestra.run.smithi077.stdout:Get:23 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [681 kB] 2022-03-08T22:52:01.884 INFO:teuthology.orchestra.run.smithi106.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [14.8 kB] 2022-03-08T22:52:01.885 INFO:teuthology.orchestra.run.smithi106.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [853 kB] 2022-03-08T22:52:01.886 INFO:teuthology.orchestra.run.smithi106.stdout:Get:19 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [536 kB] 2022-03-08T22:52:01.890 INFO:teuthology.orchestra.run.smithi077.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [909 kB] 2022-03-08T22:52:01.892 INFO:teuthology.orchestra.run.smithi106.stdout:Get:20 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [117 kB] 2022-03-08T22:52:01.893 INFO:teuthology.orchestra.run.smithi106.stdout:Get:21 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [13.2 kB] 2022-03-08T22:52:01.895 INFO:teuthology.orchestra.run.smithi106.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [23.1 kB] 2022-03-08T22:52:01.896 INFO:teuthology.orchestra.run.smithi106.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [122 kB] 2022-03-08T22:52:01.896 INFO:teuthology.orchestra.run.smithi077.stdout:Get:25 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [117 kB] 2022-03-08T22:52:01.897 INFO:teuthology.orchestra.run.smithi106.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [500 B] 2022-03-08T22:52:01.897 INFO:teuthology.orchestra.run.smithi106.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [909 kB] 2022-03-08T22:52:01.898 INFO:teuthology.orchestra.run.smithi077.stdout:Get:26 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [13.2 kB] 2022-03-08T22:52:01.898 INFO:teuthology.orchestra.run.smithi077.stdout:Get:27 http://security.ubuntu.com/ubuntu focal-security/multiverse i386 Packages [7,180 B] 2022-03-08T22:52:01.899 INFO:teuthology.orchestra.run.smithi135.stdout:Get:16 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [536 kB] 2022-03-08T22:52:01.899 INFO:teuthology.orchestra.run.smithi077.stdout:Get:28 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [20.7 kB] 2022-03-08T22:52:01.903 INFO:teuthology.orchestra.run.smithi135.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [312 kB] 2022-03-08T22:52:01.904 INFO:teuthology.orchestra.run.smithi135.stdout:Get:18 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [117 kB] 2022-03-08T22:52:01.905 INFO:teuthology.orchestra.run.smithi106.stdout:Get:26 http://security.ubuntu.com/ubuntu focal-security/multiverse i386 Packages [7,180 B] 2022-03-08T22:52:01.905 INFO:teuthology.orchestra.run.smithi106.stdout:Get:27 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [20.7 kB] 2022-03-08T22:52:01.906 INFO:teuthology.orchestra.run.smithi135.stdout:Get:19 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [13.2 kB] 2022-03-08T22:52:01.906 INFO:teuthology.orchestra.run.smithi135.stdout:Get:20 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [20.7 kB] 2022-03-08T22:52:01.906 INFO:teuthology.orchestra.run.smithi135.stdout:Get:21 http://security.ubuntu.com/ubuntu focal-security/multiverse i386 Packages [7,180 B] 2022-03-08T22:52:01.932 INFO:teuthology.orchestra.run.smithi106.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [672 kB] 2022-03-08T22:52:01.935 INFO:teuthology.orchestra.run.smithi135.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [14.8 kB] 2022-03-08T22:52:01.936 INFO:teuthology.orchestra.run.smithi135.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [23.1 kB] 2022-03-08T22:52:01.937 INFO:teuthology.orchestra.run.smithi135.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [853 kB] 2022-03-08T22:52:01.954 INFO:teuthology.orchestra.run.smithi077.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [202 kB] 2022-03-08T22:52:01.962 INFO:teuthology.orchestra.run.smithi077.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [20.2 kB] 2022-03-08T22:52:01.969 INFO:teuthology.orchestra.run.smithi077.stdout:Get:31 http://security.ubuntu.com/ubuntu focal-security/multiverse Translation-en [5,196 B] 2022-03-08T22:52:01.970 INFO:teuthology.orchestra.run.smithi077.stdout:Get:32 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 c-n-f Metadata [500 B] 2022-03-08T22:52:01.972 INFO:teuthology.orchestra.run.smithi077.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [23.8 kB] 2022-03-08T22:52:01.973 INFO:teuthology.orchestra.run.smithi077.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8,464 B] 2022-03-08T22:52:01.974 INFO:teuthology.orchestra.run.smithi077.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [7,312 B] 2022-03-08T22:52:01.974 INFO:teuthology.orchestra.run.smithi077.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [580 B] 2022-03-08T22:52:01.974 INFO:teuthology.orchestra.run.smithi077.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-backports/main i386 Packages [34.8 kB] 2022-03-08T22:52:01.979 INFO:teuthology.orchestra.run.smithi106.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [202 kB] 2022-03-08T22:52:01.983 INFO:teuthology.orchestra.run.smithi106.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [20.2 kB] 2022-03-08T22:52:01.989 INFO:teuthology.orchestra.run.smithi106.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8,464 B] 2022-03-08T22:52:01.990 INFO:teuthology.orchestra.run.smithi106.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [23.8 kB] 2022-03-08T22:52:01.990 INFO:teuthology.orchestra.run.smithi135.stdout:Get:25 http://security.ubuntu.com/ubuntu focal-security/multiverse Translation-en [5,196 B] 2022-03-08T22:52:01.991 INFO:teuthology.orchestra.run.smithi135.stdout:Get:26 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 c-n-f Metadata [500 B] 2022-03-08T22:52:01.991 INFO:teuthology.orchestra.run.smithi106.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [7,312 B] 2022-03-08T22:52:01.992 INFO:teuthology.orchestra.run.smithi106.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [580 B] 2022-03-08T22:52:01.992 INFO:teuthology.orchestra.run.smithi106.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [42.2 kB] 2022-03-08T22:52:01.996 INFO:teuthology.orchestra.run.smithi106.stdout:Get:36 http://security.ubuntu.com/ubuntu focal-security/multiverse Translation-en [5,196 B] 2022-03-08T22:52:01.996 INFO:teuthology.orchestra.run.smithi106.stdout:Get:37 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 c-n-f Metadata [500 B] 2022-03-08T22:52:01.999 INFO:teuthology.orchestra.run.smithi135.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [122 kB] 2022-03-08T22:52:02.007 INFO:teuthology.orchestra.run.smithi135.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [500 B] 2022-03-08T22:52:02.008 INFO:teuthology.orchestra.run.smithi135.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [672 kB] 2022-03-08T22:52:02.015 INFO:teuthology.orchestra.run.smithi077.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [42.2 kB] 2022-03-08T22:52:02.016 INFO:teuthology.orchestra.run.smithi077.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal-backports/main Translation-en [10.1 kB] 2022-03-08T22:52:02.037 INFO:teuthology.orchestra.run.smithi135.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [909 kB] 2022-03-08T22:52:02.042 INFO:teuthology.orchestra.run.smithi106.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal-backports/main i386 Packages [34.8 kB] 2022-03-08T22:52:02.043 INFO:teuthology.orchestra.run.smithi106.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal-backports/main Translation-en [10.1 kB] 2022-03-08T22:52:02.070 INFO:teuthology.orchestra.run.smithi135.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [202 kB] 2022-03-08T22:52:02.078 INFO:teuthology.orchestra.run.smithi135.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [20.2 kB] 2022-03-08T22:52:02.079 INFO:teuthology.orchestra.run.smithi135.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8,464 B] 2022-03-08T22:52:02.079 INFO:teuthology.orchestra.run.smithi135.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [23.8 kB] 2022-03-08T22:52:02.080 INFO:teuthology.orchestra.run.smithi135.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [7,312 B] 2022-03-08T22:52:02.099 INFO:teuthology.orchestra.run.smithi077.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [22.7 kB] 2022-03-08T22:52:02.099 INFO:teuthology.orchestra.run.smithi077.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [12.9 kB] 2022-03-08T22:52:02.100 INFO:teuthology.orchestra.run.smithi077.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal-backports/universe Translation-en [15.4 kB] 2022-03-08T22:52:02.100 INFO:teuthology.orchestra.run.smithi077.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [804 B] 2022-03-08T22:52:02.124 INFO:teuthology.orchestra.run.smithi106.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [22.7 kB] 2022-03-08T22:52:02.124 INFO:teuthology.orchestra.run.smithi106.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [12.9 kB] 2022-03-08T22:52:02.125 INFO:teuthology.orchestra.run.smithi106.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal-backports/universe Translation-en [15.4 kB] 2022-03-08T22:52:02.126 INFO:teuthology.orchestra.run.smithi106.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [804 B] 2022-03-08T22:52:02.160 INFO:teuthology.orchestra.run.smithi135.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [580 B] 2022-03-08T22:52:02.161 INFO:teuthology.orchestra.run.smithi135.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-backports/main i386 Packages [34.8 kB] 2022-03-08T22:52:02.162 INFO:teuthology.orchestra.run.smithi135.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [42.2 kB] 2022-03-08T22:52:02.163 INFO:teuthology.orchestra.run.smithi135.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal-backports/main Translation-en [10.1 kB] 2022-03-08T22:52:02.164 INFO:teuthology.orchestra.run.smithi135.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [22.7 kB] 2022-03-08T22:52:02.242 INFO:teuthology.orchestra.run.smithi135.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [12.9 kB] 2022-03-08T22:52:02.243 INFO:teuthology.orchestra.run.smithi135.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal-backports/universe Translation-en [15.4 kB] 2022-03-08T22:52:02.244 INFO:teuthology.orchestra.run.smithi135.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [804 B] 2022-03-08T22:52:04.668 INFO:teuthology.orchestra.run.smithi106.stdout:Fetched 10.2 MB in 2s (4,615 kB/s) 2022-03-08T22:52:04.730 INFO:teuthology.orchestra.run.smithi135.stdout:Fetched 10.2 MB in 2s (4,734 kB/s) 2022-03-08T22:52:04.746 INFO:teuthology.orchestra.run.smithi077.stdout:Fetched 10.2 MB in 2s (4,589 kB/s) 2022-03-08T22:52:05.825 INFO:teuthology.orchestra.run.smithi106.stdout:Reading package lists... 2022-03-08T22:52:05.849 DEBUG:teuthology.orchestra.run.smithi106:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic-hwe-20.04 2022-03-08T22:52:05.882 INFO:teuthology.orchestra.run.smithi077.stdout:Reading package lists... 2022-03-08T22:52:05.903 DEBUG:teuthology.orchestra.run.smithi077:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic-hwe-20.04 2022-03-08T22:52:05.912 INFO:teuthology.orchestra.run.smithi106.stdout:Reading package lists... 2022-03-08T22:52:05.939 INFO:teuthology.orchestra.run.smithi135.stdout:Reading package lists... 2022-03-08T22:52:05.963 DEBUG:teuthology.orchestra.run.smithi135:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic-hwe-20.04 2022-03-08T22:52:05.968 INFO:teuthology.orchestra.run.smithi077.stdout:Reading package lists... 2022-03-08T22:52:06.034 INFO:teuthology.orchestra.run.smithi135.stdout:Reading package lists... 2022-03-08T22:52:06.066 INFO:teuthology.orchestra.run.smithi106.stdout:Building dependency tree... 2022-03-08T22:52:06.067 INFO:teuthology.orchestra.run.smithi106.stdout:Reading state information... 2022-03-08T22:52:06.143 INFO:teuthology.orchestra.run.smithi077.stdout:Building dependency tree... 2022-03-08T22:52:06.144 INFO:teuthology.orchestra.run.smithi077.stdout:Reading state information... 2022-03-08T22:52:06.198 INFO:teuthology.orchestra.run.smithi106.stdout:The following additional packages will be installed: 2022-03-08T22:52:06.198 INFO:teuthology.orchestra.run.smithi106.stdout: libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 2022-03-08T22:52:06.199 INFO:teuthology.orchestra.run.smithi106.stdout: libusbmuxd6 linux-image-5.13.0-35-generic linux-modules-5.13.0-35-generic 2022-03-08T22:52:06.199 INFO:teuthology.orchestra.run.smithi106.stdout: linux-modules-extra-5.13.0-35-generic thermald upower usbmuxd 2022-03-08T22:52:06.199 INFO:teuthology.orchestra.run.smithi106.stdout:Suggested packages: 2022-03-08T22:52:06.200 INFO:teuthology.orchestra.run.smithi106.stdout: libusbmuxd-tools fdutils linux-doc | linux-hwe-5.13-source-5.13.0 2022-03-08T22:52:06.200 INFO:teuthology.orchestra.run.smithi106.stdout: linux-hwe-5.13-tools linux-headers-5.13.0-35-generic 2022-03-08T22:52:06.208 INFO:teuthology.orchestra.run.smithi135.stdout:Building dependency tree... 2022-03-08T22:52:06.209 INFO:teuthology.orchestra.run.smithi135.stdout:Reading state information... 2022-03-08T22:52:06.258 INFO:teuthology.orchestra.run.smithi106.stdout:The following NEW packages will be installed: 2022-03-08T22:52:06.259 INFO:teuthology.orchestra.run.smithi106.stdout: libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 2022-03-08T22:52:06.260 INFO:teuthology.orchestra.run.smithi106.stdout: libusbmuxd6 linux-image-5.13.0-35-generic linux-image-generic-hwe-20.04 2022-03-08T22:52:06.260 INFO:teuthology.orchestra.run.smithi106.stdout: linux-modules-5.13.0-35-generic linux-modules-extra-5.13.0-35-generic 2022-03-08T22:52:06.260 INFO:teuthology.orchestra.run.smithi106.stdout: thermald upower usbmuxd 2022-03-08T22:52:06.276 INFO:teuthology.orchestra.run.smithi077.stdout:The following additional packages will be installed: 2022-03-08T22:52:06.277 INFO:teuthology.orchestra.run.smithi077.stdout: libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 2022-03-08T22:52:06.277 INFO:teuthology.orchestra.run.smithi077.stdout: libusbmuxd6 linux-image-5.13.0-35-generic linux-modules-5.13.0-35-generic 2022-03-08T22:52:06.277 INFO:teuthology.orchestra.run.smithi077.stdout: linux-modules-extra-5.13.0-35-generic thermald upower usbmuxd 2022-03-08T22:52:06.278 INFO:teuthology.orchestra.run.smithi077.stdout:Suggested packages: 2022-03-08T22:52:06.278 INFO:teuthology.orchestra.run.smithi077.stdout: libusbmuxd-tools fdutils linux-doc | linux-hwe-5.13-source-5.13.0 2022-03-08T22:52:06.278 INFO:teuthology.orchestra.run.smithi077.stdout: linux-hwe-5.13-tools linux-headers-5.13.0-35-generic 2022-03-08T22:52:06.326 INFO:teuthology.orchestra.run.smithi077.stdout:The following NEW packages will be installed: 2022-03-08T22:52:06.327 INFO:teuthology.orchestra.run.smithi077.stdout: libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 2022-03-08T22:52:06.327 INFO:teuthology.orchestra.run.smithi077.stdout: libusbmuxd6 linux-image-5.13.0-35-generic linux-image-generic-hwe-20.04 2022-03-08T22:52:06.327 INFO:teuthology.orchestra.run.smithi077.stdout: linux-modules-5.13.0-35-generic linux-modules-extra-5.13.0-35-generic 2022-03-08T22:52:06.328 INFO:teuthology.orchestra.run.smithi077.stdout: thermald upower usbmuxd 2022-03-08T22:52:06.354 INFO:teuthology.orchestra.run.smithi135.stdout:The following additional packages will be installed: 2022-03-08T22:52:06.354 INFO:teuthology.orchestra.run.smithi135.stdout: libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 2022-03-08T22:52:06.355 INFO:teuthology.orchestra.run.smithi135.stdout: libusbmuxd6 linux-image-5.13.0-35-generic linux-modules-5.13.0-35-generic 2022-03-08T22:52:06.355 INFO:teuthology.orchestra.run.smithi135.stdout: linux-modules-extra-5.13.0-35-generic thermald upower usbmuxd 2022-03-08T22:52:06.356 INFO:teuthology.orchestra.run.smithi135.stdout:Suggested packages: 2022-03-08T22:52:06.356 INFO:teuthology.orchestra.run.smithi135.stdout: libusbmuxd-tools fdutils linux-doc | linux-hwe-5.13-source-5.13.0 2022-03-08T22:52:06.356 INFO:teuthology.orchestra.run.smithi135.stdout: linux-hwe-5.13-tools linux-headers-5.13.0-35-generic 2022-03-08T22:52:06.401 INFO:teuthology.orchestra.run.smithi135.stdout:The following NEW packages will be installed: 2022-03-08T22:52:06.402 INFO:teuthology.orchestra.run.smithi135.stdout: libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 2022-03-08T22:52:06.402 INFO:teuthology.orchestra.run.smithi135.stdout: libusbmuxd6 linux-image-5.13.0-35-generic linux-image-generic-hwe-20.04 2022-03-08T22:52:06.402 INFO:teuthology.orchestra.run.smithi135.stdout: linux-modules-5.13.0-35-generic linux-modules-extra-5.13.0-35-generic 2022-03-08T22:52:06.402 INFO:teuthology.orchestra.run.smithi135.stdout: thermald upower usbmuxd 2022-03-08T22:52:06.442 INFO:teuthology.orchestra.run.smithi106.stdout:0 upgraded, 13 newly installed, 0 to remove and 92 not upgraded. 2022-03-08T22:52:06.443 INFO:teuthology.orchestra.run.smithi106.stdout:Need to get 80.1 MB of archives. 2022-03-08T22:52:06.443 INFO:teuthology.orchestra.run.smithi106.stdout:After this operation, 409 MB of additional disk space will be used. 2022-03-08T22:52:06.443 INFO:teuthology.orchestra.run.smithi106.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 libdbus-glib-1-2 amd64 0.110-5fakssync1 [59.1 kB] 2022-03-08T22:52:06.508 INFO:teuthology.orchestra.run.smithi077.stdout:0 upgraded, 13 newly installed, 0 to remove and 92 not upgraded. 2022-03-08T22:52:06.508 INFO:teuthology.orchestra.run.smithi077.stdout:Need to get 80.1 MB of archives. 2022-03-08T22:52:06.508 INFO:teuthology.orchestra.run.smithi077.stdout:After this operation, 409 MB of additional disk space will be used. 2022-03-08T22:52:06.509 INFO:teuthology.orchestra.run.smithi077.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 libdbus-glib-1-2 amd64 0.110-5fakssync1 [59.1 kB] 2022-03-08T22:52:06.593 INFO:teuthology.orchestra.run.smithi135.stdout:0 upgraded, 13 newly installed, 0 to remove and 92 not upgraded. 2022-03-08T22:52:06.594 INFO:teuthology.orchestra.run.smithi135.stdout:Need to get 80.1 MB of archives. 2022-03-08T22:52:06.594 INFO:teuthology.orchestra.run.smithi135.stdout:After this operation, 409 MB of additional disk space will be used. 2022-03-08T22:52:06.595 INFO:teuthology.orchestra.run.smithi135.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 libdbus-glib-1-2 amd64 0.110-5fakssync1 [59.1 kB] 2022-03-08T22:52:06.713 INFO:teuthology.orchestra.run.smithi106.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 libplist3 amd64 2.1.0-4build2 [31.6 kB] 2022-03-08T22:52:06.741 INFO:teuthology.orchestra.run.smithi106.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 libusbmuxd6 amd64 2.0.1-2 [19.1 kB] 2022-03-08T22:52:06.742 INFO:teuthology.orchestra.run.smithi106.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 libimobiledevice6 amd64 1.2.1~git20191129.9f79242-1build1 [65.2 kB] 2022-03-08T22:52:06.779 INFO:teuthology.orchestra.run.smithi077.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 libplist3 amd64 2.1.0-4build2 [31.6 kB] 2022-03-08T22:52:06.813 INFO:teuthology.orchestra.run.smithi077.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 libusbmuxd6 amd64 2.0.1-2 [19.1 kB] 2022-03-08T22:52:06.829 INFO:teuthology.orchestra.run.smithi106.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 libupower-glib3 amd64 0.99.11-1build2 [43.2 kB] 2022-03-08T22:52:06.830 INFO:teuthology.orchestra.run.smithi106.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.13.0-35-generic amd64 5.13.0-35.40~20.04.1 [18.3 MB] 2022-03-08T22:52:06.836 INFO:teuthology.orchestra.run.smithi077.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 libimobiledevice6 amd64 1.2.1~git20191129.9f79242-1build1 [65.2 kB] 2022-03-08T22:52:06.874 INFO:teuthology.orchestra.run.smithi135.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 libplist3 amd64 2.1.0-4build2 [31.6 kB] 2022-03-08T22:52:06.894 INFO:teuthology.orchestra.run.smithi135.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 libusbmuxd6 amd64 2.0.1-2 [19.1 kB] 2022-03-08T22:52:06.906 INFO:teuthology.orchestra.run.smithi077.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 libupower-glib3 amd64 0.99.11-1build2 [43.2 kB] 2022-03-08T22:52:06.915 INFO:teuthology.orchestra.run.smithi135.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 libimobiledevice6 amd64 1.2.1~git20191129.9f79242-1build1 [65.2 kB] 2022-03-08T22:52:06.935 INFO:teuthology.orchestra.run.smithi077.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.13.0-35-generic amd64 5.13.0-35.40~20.04.1 [18.3 MB] 2022-03-08T22:52:06.996 INFO:teuthology.orchestra.run.smithi135.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 libupower-glib3 amd64 0.99.11-1build2 [43.2 kB] 2022-03-08T22:52:07.015 INFO:teuthology.orchestra.run.smithi135.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.13.0-35-generic amd64 5.13.0-35.40~20.04.1 [18.3 MB] 2022-03-08T22:52:07.729 INFO:teuthology.orchestra.run.smithi106.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.13.0-35-generic amd64 5.13.0-35.40~20.04.1 [10.0 MB] 2022-03-08T22:52:07.815 INFO:teuthology.orchestra.run.smithi077.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.13.0-35-generic amd64 5.13.0-35.40~20.04.1 [10.0 MB] 2022-03-08T22:52:07.957 INFO:teuthology.orchestra.run.smithi135.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.13.0-35-generic amd64 5.13.0-35.40~20.04.1 [10.0 MB] 2022-03-08T22:52:08.035 INFO:teuthology.orchestra.run.smithi106.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.13.0-35-generic amd64 5.13.0-35.40~20.04.1 [51.1 MB] 2022-03-08T22:52:08.161 INFO:teuthology.orchestra.run.smithi077.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.13.0-35-generic amd64 5.13.0-35.40~20.04.1 [51.1 MB] 2022-03-08T22:52:08.316 INFO:teuthology.orchestra.run.smithi135.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.13.0-35-generic amd64 5.13.0-35.40~20.04.1 [51.1 MB] 2022-03-08T22:52:09.917 INFO:teuthology.orchestra.run.smithi106.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic-hwe-20.04 amd64 5.13.0.35.40~20.04.20 [2,632 B] 2022-03-08T22:52:09.918 INFO:teuthology.orchestra.run.smithi106.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libevdev2 amd64 1.9.0+dfsg-1ubuntu0.1 [31.6 kB] 2022-03-08T22:52:09.918 INFO:teuthology.orchestra.run.smithi106.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 thermald amd64 1.9.1-1ubuntu0.6 [233 kB] 2022-03-08T22:52:09.924 INFO:teuthology.orchestra.run.smithi106.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/main amd64 upower amd64 0.99.11-1build2 [104 kB] 2022-03-08T22:52:09.926 INFO:teuthology.orchestra.run.smithi106.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 usbmuxd amd64 1.1.1~git20191130.9af2b12-1 [38.4 kB] 2022-03-08T22:52:10.159 INFO:teuthology.orchestra.run.smithi077.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic-hwe-20.04 amd64 5.13.0.35.40~20.04.20 [2,632 B] 2022-03-08T22:52:10.160 INFO:teuthology.orchestra.run.smithi077.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libevdev2 amd64 1.9.0+dfsg-1ubuntu0.1 [31.6 kB] 2022-03-08T22:52:10.160 INFO:teuthology.orchestra.run.smithi077.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 thermald amd64 1.9.1-1ubuntu0.6 [233 kB] 2022-03-08T22:52:10.165 INFO:teuthology.orchestra.run.smithi077.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/main amd64 upower amd64 0.99.11-1build2 [104 kB] 2022-03-08T22:52:10.167 INFO:teuthology.orchestra.run.smithi077.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 usbmuxd amd64 1.1.1~git20191130.9af2b12-1 [38.4 kB] 2022-03-08T22:52:10.340 INFO:teuthology.orchestra.run.smithi106.stdout:Fetched 80.1 MB in 4s (21.9 MB/s) 2022-03-08T22:52:10.359 INFO:teuthology.orchestra.run.smithi135.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic-hwe-20.04 amd64 5.13.0.35.40~20.04.20 [2,632 B] 2022-03-08T22:52:10.360 INFO:teuthology.orchestra.run.smithi135.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libevdev2 amd64 1.9.0+dfsg-1ubuntu0.1 [31.6 kB] 2022-03-08T22:52:10.360 INFO:teuthology.orchestra.run.smithi135.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 thermald amd64 1.9.1-1ubuntu0.6 [233 kB] 2022-03-08T22:52:10.366 INFO:teuthology.orchestra.run.smithi135.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/main amd64 upower amd64 0.99.11-1build2 [104 kB] 2022-03-08T22:52:10.368 INFO:teuthology.orchestra.run.smithi135.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 usbmuxd amd64 1.1.1~git20191130.9af2b12-1 [38.4 kB] 2022-03-08T22:52:10.555 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package libdbus-glib-1-2:amd64. 2022-03-08T22:52:10.601 INFO:teuthology.orchestra.run.smithi077.stdout:Fetched 80.1 MB in 4s (20.9 MB/s) 2022-03-08T22:52:10.785 INFO:teuthology.orchestra.run.smithi135.stdout:Fetched 80.1 MB in 4s (20.2 MB/s) 2022-03-08T22:52:10.808 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package libdbus-glib-1-2:amd64. 2022-03-08T22:52:10.992 INFO:teuthology.orchestra.run.smithi135.stdout:Selecting previously unselected package libdbus-glib-1-2:amd64. 2022-03-08T22:52:11.964 INFO:teuthology.orchestra.run.smithi106.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 102042 files and directories currently installed.) 2022-03-08T22:52:11.968 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../00-libdbus-glib-1-2_0.110-5fakssync1_amd64.deb ... 2022-03-08T22:52:12.101 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking libdbus-glib-1-2:amd64 (0.110-5fakssync1) ... 2022-03-08T22:52:12.262 INFO:teuthology.orchestra.run.smithi077.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 102042 files and directories currently installed.) 2022-03-08T22:52:12.267 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../00-libdbus-glib-1-2_0.110-5fakssync1_amd64.deb ... 2022-03-08T22:52:12.380 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking libdbus-glib-1-2:amd64 (0.110-5fakssync1) ... 2022-03-08T22:52:12.431 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package libplist3:amd64. 2022-03-08T22:52:12.446 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../01-libplist3_2.1.0-4build2_amd64.deb ... 2022-03-08T22:52:12.482 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking libplist3:amd64 (2.1.0-4build2) ... 2022-03-08T22:52:12.534 INFO:teuthology.orchestra.run.smithi135.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 102042 files and directories currently installed.) 2022-03-08T22:52:12.539 INFO:teuthology.orchestra.run.smithi135.stdout:Preparing to unpack .../00-libdbus-glib-1-2_0.110-5fakssync1_amd64.deb ... 2022-03-08T22:52:12.672 INFO:teuthology.orchestra.run.smithi135.stdout:Unpacking libdbus-glib-1-2:amd64 (0.110-5fakssync1) ... 2022-03-08T22:52:12.742 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package libplist3:amd64. 2022-03-08T22:52:12.756 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../01-libplist3_2.1.0-4build2_amd64.deb ... 2022-03-08T22:52:12.784 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package libusbmuxd6:amd64. 2022-03-08T22:52:12.799 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../02-libusbmuxd6_2.0.1-2_amd64.deb ... 2022-03-08T22:52:12.800 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking libplist3:amd64 (2.1.0-4build2) ... 2022-03-08T22:52:12.842 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking libusbmuxd6:amd64 (2.0.1-2) ... 2022-03-08T22:52:13.076 INFO:teuthology.orchestra.run.smithi135.stdout:Selecting previously unselected package libplist3:amd64. 2022-03-08T22:52:13.089 INFO:teuthology.orchestra.run.smithi135.stdout:Preparing to unpack .../01-libplist3_2.1.0-4build2_amd64.deb ... 2022-03-08T22:52:13.134 INFO:teuthology.orchestra.run.smithi135.stdout:Unpacking libplist3:amd64 (2.1.0-4build2) ... 2022-03-08T22:52:13.144 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package libusbmuxd6:amd64. 2022-03-08T22:52:13.157 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../02-libusbmuxd6_2.0.1-2_amd64.deb ... 2022-03-08T22:52:13.187 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package libimobiledevice6:amd64. 2022-03-08T22:52:13.194 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking libusbmuxd6:amd64 (2.0.1-2) ... 2022-03-08T22:52:13.200 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../03-libimobiledevice6_1.2.1~git20191129.9f79242-1build1_amd64.deb ... 2022-03-08T22:52:13.245 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking libimobiledevice6:amd64 (1.2.1~git20191129.9f79242-1build1) ... 2022-03-08T22:52:13.412 INFO:teuthology.orchestra.run.smithi135.stdout:Selecting previously unselected package libusbmuxd6:amd64. 2022-03-08T22:52:13.425 INFO:teuthology.orchestra.run.smithi135.stdout:Preparing to unpack .../02-libusbmuxd6_2.0.1-2_amd64.deb ... 2022-03-08T22:52:13.462 INFO:teuthology.orchestra.run.smithi135.stdout:Unpacking libusbmuxd6:amd64 (2.0.1-2) ... 2022-03-08T22:52:13.505 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package libimobiledevice6:amd64. 2022-03-08T22:52:13.518 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../03-libimobiledevice6_1.2.1~git20191129.9f79242-1build1_amd64.deb ... 2022-03-08T22:52:13.555 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking libimobiledevice6:amd64 (1.2.1~git20191129.9f79242-1build1) ... 2022-03-08T22:52:13.615 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package libupower-glib3:amd64. 2022-03-08T22:52:13.628 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../04-libupower-glib3_0.99.11-1build2_amd64.deb ... 2022-03-08T22:52:13.673 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking libupower-glib3:amd64 (0.99.11-1build2) ... 2022-03-08T22:52:13.807 INFO:teuthology.orchestra.run.smithi135.stdout:Selecting previously unselected package libimobiledevice6:amd64. 2022-03-08T22:52:13.819 INFO:teuthology.orchestra.run.smithi135.stdout:Preparing to unpack .../03-libimobiledevice6_1.2.1~git20191129.9f79242-1build1_amd64.deb ... 2022-03-08T22:52:13.857 INFO:teuthology.orchestra.run.smithi135.stdout:Unpacking libimobiledevice6:amd64 (1.2.1~git20191129.9f79242-1build1) ... 2022-03-08T22:52:13.899 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package libupower-glib3:amd64. 2022-03-08T22:52:13.915 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../04-libupower-glib3_0.99.11-1build2_amd64.deb ... 2022-03-08T22:52:13.951 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package linux-modules-5.13.0-35-generic. 2022-03-08T22:52:13.957 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking libupower-glib3:amd64 (0.99.11-1build2) ... 2022-03-08T22:52:13.964 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../05-linux-modules-5.13.0-35-generic_5.13.0-35.40~20.04.1_amd64.deb ... 2022-03-08T22:52:14.000 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking linux-modules-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-08T22:52:14.218 INFO:teuthology.orchestra.run.smithi135.stdout:Selecting previously unselected package libupower-glib3:amd64. 2022-03-08T22:52:14.234 INFO:teuthology.orchestra.run.smithi135.stdout:Preparing to unpack .../04-libupower-glib3_0.99.11-1build2_amd64.deb ... 2022-03-08T22:52:14.276 INFO:teuthology.orchestra.run.smithi135.stdout:Unpacking libupower-glib3:amd64 (0.99.11-1build2) ... 2022-03-08T22:52:14.285 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package linux-modules-5.13.0-35-generic. 2022-03-08T22:52:14.300 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../05-linux-modules-5.13.0-35-generic_5.13.0-35.40~20.04.1_amd64.deb ... 2022-03-08T22:52:14.343 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking linux-modules-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-08T22:52:14.562 INFO:teuthology.orchestra.run.smithi135.stdout:Selecting previously unselected package linux-modules-5.13.0-35-generic. 2022-03-08T22:52:14.575 INFO:teuthology.orchestra.run.smithi135.stdout:Preparing to unpack .../05-linux-modules-5.13.0-35-generic_5.13.0-35.40~20.04.1_amd64.deb ... 2022-03-08T22:52:14.604 INFO:teuthology.orchestra.run.smithi135.stdout:Unpacking linux-modules-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-08T22:52:16.225 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package linux-image-5.13.0-35-generic. 2022-03-08T22:52:16.238 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../06-linux-image-5.13.0-35-generic_5.13.0-35.40~20.04.1_amd64.deb ... 2022-03-08T22:52:16.325 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking linux-image-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-08T22:52:16.777 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package linux-image-5.13.0-35-generic. 2022-03-08T22:52:16.793 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../06-linux-image-5.13.0-35-generic_5.13.0-35.40~20.04.1_amd64.deb ... 2022-03-08T22:52:16.812 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package linux-modules-extra-5.13.0-35-generic. 2022-03-08T22:52:16.825 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../07-linux-modules-extra-5.13.0-35-generic_5.13.0-35.40~20.04.1_amd64.deb ... 2022-03-08T22:52:16.861 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking linux-modules-extra-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-08T22:52:16.880 INFO:teuthology.orchestra.run.smithi135.stdout:Selecting previously unselected package linux-image-5.13.0-35-generic. 2022-03-08T22:52:16.890 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking linux-image-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-08T22:52:16.894 INFO:teuthology.orchestra.run.smithi135.stdout:Preparing to unpack .../06-linux-image-5.13.0-35-generic_5.13.0-35.40~20.04.1_amd64.deb ... 2022-03-08T22:52:17.007 INFO:teuthology.orchestra.run.smithi135.stdout:Unpacking linux-image-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-08T22:52:17.397 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package linux-modules-extra-5.13.0-35-generic. 2022-03-08T22:52:17.410 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../07-linux-modules-extra-5.13.0-35-generic_5.13.0-35.40~20.04.1_amd64.deb ... 2022-03-08T22:52:17.463 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking linux-modules-extra-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-08T22:52:17.536 INFO:teuthology.orchestra.run.smithi135.stdout:Selecting previously unselected package linux-modules-extra-5.13.0-35-generic. 2022-03-08T22:52:17.549 INFO:teuthology.orchestra.run.smithi135.stdout:Preparing to unpack .../07-linux-modules-extra-5.13.0-35-generic_5.13.0-35.40~20.04.1_amd64.deb ... 2022-03-08T22:52:17.592 INFO:teuthology.orchestra.run.smithi135.stdout:Unpacking linux-modules-extra-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-08T22:52:22.827 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package linux-image-generic-hwe-20.04. 2022-03-08T22:52:22.842 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../08-linux-image-generic-hwe-20.04_5.13.0.35.40~20.04.20_amd64.deb ... 2022-03-08T22:52:22.876 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking linux-image-generic-hwe-20.04 (5.13.0.35.40~20.04.20) ... 2022-03-08T22:52:23.184 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package libevdev2:amd64. 2022-03-08T22:52:23.198 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../09-libevdev2_1.9.0+dfsg-1ubuntu0.1_amd64.deb ... 2022-03-08T22:52:23.242 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking libevdev2:amd64 (1.9.0+dfsg-1ubuntu0.1) ... 2022-03-08T22:52:23.386 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package linux-image-generic-hwe-20.04. 2022-03-08T22:52:23.401 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../08-linux-image-generic-hwe-20.04_5.13.0.35.40~20.04.20_amd64.deb ... 2022-03-08T22:52:23.446 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking linux-image-generic-hwe-20.04 (5.13.0.35.40~20.04.20) ... 2022-03-08T22:52:23.620 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package thermald. 2022-03-08T22:52:23.636 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../10-thermald_1.9.1-1ubuntu0.6_amd64.deb ... 2022-03-08T22:52:23.678 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking thermald (1.9.1-1ubuntu0.6) ... 2022-03-08T22:52:23.765 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package libevdev2:amd64. 2022-03-08T22:52:23.781 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../09-libevdev2_1.9.0+dfsg-1ubuntu0.1_amd64.deb ... 2022-03-08T22:52:23.824 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking libevdev2:amd64 (1.9.0+dfsg-1ubuntu0.1) ... 2022-03-08T22:52:24.068 INFO:teuthology.orchestra.run.smithi135.stdout:Selecting previously unselected package linux-image-generic-hwe-20.04. 2022-03-08T22:52:24.086 INFO:teuthology.orchestra.run.smithi135.stdout:Preparing to unpack .../08-linux-image-generic-hwe-20.04_5.13.0.35.40~20.04.20_amd64.deb ... 2022-03-08T22:52:24.126 INFO:teuthology.orchestra.run.smithi135.stdout:Unpacking linux-image-generic-hwe-20.04 (5.13.0.35.40~20.04.20) ... 2022-03-08T22:52:24.201 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package thermald. 2022-03-08T22:52:24.219 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../10-thermald_1.9.1-1ubuntu0.6_amd64.deb ... 2022-03-08T22:52:24.227 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package upower. 2022-03-08T22:52:24.241 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../11-upower_0.99.11-1build2_amd64.deb ... 2022-03-08T22:52:24.260 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking thermald (1.9.1-1ubuntu0.6) ... 2022-03-08T22:52:24.296 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking upower (0.99.11-1build2) ... 2022-03-08T22:52:24.421 INFO:teuthology.orchestra.run.smithi135.stdout:Selecting previously unselected package libevdev2:amd64. 2022-03-08T22:52:24.434 INFO:teuthology.orchestra.run.smithi135.stdout:Preparing to unpack .../09-libevdev2_1.9.0+dfsg-1ubuntu0.1_amd64.deb ... 2022-03-08T22:52:24.479 INFO:teuthology.orchestra.run.smithi135.stdout:Unpacking libevdev2:amd64 (1.9.0+dfsg-1ubuntu0.1) ... 2022-03-08T22:52:24.711 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package usbmuxd. 2022-03-08T22:52:24.727 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../12-usbmuxd_1.1.1~git20191130.9af2b12-1_amd64.deb ... 2022-03-08T22:52:24.769 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking usbmuxd (1.1.1~git20191130.9af2b12-1) ... 2022-03-08T22:52:24.800 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package upower. 2022-03-08T22:52:24.814 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../11-upower_0.99.11-1build2_amd64.deb ... 2022-03-08T22:52:24.844 INFO:teuthology.orchestra.run.smithi135.stdout:Selecting previously unselected package thermald. 2022-03-08T22:52:24.857 INFO:teuthology.orchestra.run.smithi135.stdout:Preparing to unpack .../10-thermald_1.9.1-1ubuntu0.6_amd64.deb ... 2022-03-08T22:52:24.869 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking upower (0.99.11-1build2) ... 2022-03-08T22:52:24.893 INFO:teuthology.orchestra.run.smithi135.stdout:Unpacking thermald (1.9.1-1ubuntu0.6) ... 2022-03-08T22:52:25.058 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up libplist3:amd64 (2.1.0-4build2) ... 2022-03-08T22:52:25.183 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up libdbus-glib-1-2:amd64 (0.110-5fakssync1) ... 2022-03-08T22:52:25.309 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package usbmuxd. 2022-03-08T22:52:25.309 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up libusbmuxd6:amd64 (2.0.1-2) ... 2022-03-08T22:52:25.322 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../12-usbmuxd_1.1.1~git20191130.9af2b12-1_amd64.deb ... 2022-03-08T22:52:25.367 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking usbmuxd (1.1.1~git20191130.9af2b12-1) ... 2022-03-08T22:52:25.445 INFO:teuthology.orchestra.run.smithi135.stdout:Selecting previously unselected package upower. 2022-03-08T22:52:25.451 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up libupower-glib3:amd64 (0.99.11-1build2) ... 2022-03-08T22:52:25.458 INFO:teuthology.orchestra.run.smithi135.stdout:Preparing to unpack .../11-upower_0.99.11-1build2_amd64.deb ... 2022-03-08T22:52:25.515 INFO:teuthology.orchestra.run.smithi135.stdout:Unpacking upower (0.99.11-1build2) ... 2022-03-08T22:52:25.561 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up libimobiledevice6:amd64 (1.2.1~git20191129.9f79242-1build1) ... 2022-03-08T22:52:25.659 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up libplist3:amd64 (2.1.0-4build2) ... 2022-03-08T22:52:25.678 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up libevdev2:amd64 (1.9.0+dfsg-1ubuntu0.1) ... 2022-03-08T22:52:25.763 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up libdbus-glib-1-2:amd64 (0.110-5fakssync1) ... 2022-03-08T22:52:25.804 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up upower (0.99.11-1build2) ... 2022-03-08T22:52:25.881 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up libusbmuxd6:amd64 (2.0.1-2) ... 2022-03-08T22:52:25.956 INFO:teuthology.orchestra.run.smithi135.stdout:Selecting previously unselected package usbmuxd. 2022-03-08T22:52:25.970 INFO:teuthology.orchestra.run.smithi135.stdout:Preparing to unpack .../12-usbmuxd_1.1.1~git20191130.9af2b12-1_amd64.deb ... 2022-03-08T22:52:26.007 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up libupower-glib3:amd64 (0.99.11-1build2) ... 2022-03-08T22:52:26.014 INFO:teuthology.orchestra.run.smithi135.stdout:Unpacking usbmuxd (1.1.1~git20191130.9af2b12-1) ... 2022-03-08T22:52:26.133 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up libimobiledevice6:amd64 (1.2.1~git20191129.9f79242-1build1) ... 2022-03-08T22:52:26.218 INFO:teuthology.orchestra.run.smithi106.stdout:upower.service is a disabled or a static unit, not starting it. 2022-03-08T22:52:26.259 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up libevdev2:amd64 (1.9.0+dfsg-1ubuntu0.1) ... 2022-03-08T22:52:26.290 INFO:teuthology.orchestra.run.smithi135.stdout:Setting up libplist3:amd64 (2.1.0-4build2) ... 2022-03-08T22:52:26.300 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up usbmuxd (1.1.1~git20191130.9af2b12-1) ... 2022-03-08T22:52:26.407 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up upower (0.99.11-1build2) ... 2022-03-08T22:52:26.428 INFO:teuthology.orchestra.run.smithi135.stdout:Setting up libdbus-glib-1-2:amd64 (0.110-5fakssync1) ... 2022-03-08T22:52:26.482 INFO:teuthology.orchestra.run.smithi106.stdout:Warning: The home dir /var/lib/usbmux you specified can't be accessed: No such file or directory 2022-03-08T22:52:26.494 INFO:teuthology.orchestra.run.smithi106.stdout:Adding system user `usbmux' (UID 114) ... 2022-03-08T22:52:26.494 INFO:teuthology.orchestra.run.smithi106.stdout:Adding new user `usbmux' (UID 114) with group `plugdev' ... 2022-03-08T22:52:26.546 INFO:teuthology.orchestra.run.smithi135.stdout:Setting up libusbmuxd6:amd64 (2.0.1-2) ... 2022-03-08T22:52:26.673 INFO:teuthology.orchestra.run.smithi135.stdout:Setting up libupower-glib3:amd64 (0.99.11-1build2) ... 2022-03-08T22:52:26.808 INFO:teuthology.orchestra.run.smithi135.stdout:Setting up libimobiledevice6:amd64 (1.2.1~git20191129.9f79242-1build1) ... 2022-03-08T22:52:26.864 INFO:teuthology.orchestra.run.smithi077.stdout:upower.service is a disabled or a static unit, not starting it. 2022-03-08T22:52:26.934 INFO:teuthology.orchestra.run.smithi135.stdout:Setting up libevdev2:amd64 (1.9.0+dfsg-1ubuntu0.1) ... 2022-03-08T22:52:26.953 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up usbmuxd (1.1.1~git20191130.9af2b12-1) ... 2022-03-08T22:52:26.982 INFO:teuthology.orchestra.run.smithi106.stdout:Not creating home directory `/var/lib/usbmux'. 2022-03-08T22:52:27.077 INFO:teuthology.orchestra.run.smithi135.stdout:Setting up upower (0.99.11-1build2) ... 2022-03-08T22:52:27.135 INFO:teuthology.orchestra.run.smithi077.stdout:Warning: The home dir /var/lib/usbmux you specified can't be accessed: No such file or directory 2022-03-08T22:52:27.149 INFO:teuthology.orchestra.run.smithi077.stdout:Adding system user `usbmux' (UID 114) ... 2022-03-08T22:52:27.149 INFO:teuthology.orchestra.run.smithi077.stdout:Adding new user `usbmux' (UID 114) with group `plugdev' ... 2022-03-08T22:52:27.206 INFO:teuthology.orchestra.run.smithi106.stdout:usbmuxd.service is a disabled or a static unit, not starting it. 2022-03-08T22:52:27.244 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up thermald (1.9.1-1ubuntu0.6) ... 2022-03-08T22:52:27.545 INFO:teuthology.orchestra.run.smithi135.stdout:upower.service is a disabled or a static unit, not starting it. 2022-03-08T22:52:27.549 INFO:teuthology.orchestra.run.smithi106.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.thermald.service -> /lib/systemd/system/thermald.service. 2022-03-08T22:52:27.549 INFO:teuthology.orchestra.run.smithi106.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/thermald.service -> /lib/systemd/system/thermald.service. 2022-03-08T22:52:27.553 INFO:teuthology.orchestra.run.smithi077.stdout:Not creating home directory `/var/lib/usbmux'. 2022-03-08T22:52:27.587 INFO:teuthology.orchestra.run.smithi135.stdout:Setting up usbmuxd (1.1.1~git20191130.9af2b12-1) ... 2022-03-08T22:52:27.772 INFO:teuthology.orchestra.run.smithi135.stdout:Warning: The home dir /var/lib/usbmux you specified can't be accessed: No such file or directory 2022-03-08T22:52:27.781 INFO:teuthology.orchestra.run.smithi135.stdout:Adding system user `usbmux' (UID 114) ... 2022-03-08T22:52:27.781 INFO:teuthology.orchestra.run.smithi135.stdout:Adding new user `usbmux' (UID 114) with group `plugdev' ... 2022-03-08T22:52:27.793 INFO:teuthology.orchestra.run.smithi077.stdout:usbmuxd.service is a disabled or a static unit, not starting it. 2022-03-08T22:52:27.833 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up thermald (1.9.1-1ubuntu0.6) ... 2022-03-08T22:52:28.065 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up linux-image-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-08T22:52:28.140 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.thermald.service -> /lib/systemd/system/thermald.service. 2022-03-08T22:52:28.141 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/thermald.service -> /lib/systemd/system/thermald.service. 2022-03-08T22:52:28.272 INFO:teuthology.orchestra.run.smithi135.stdout:Not creating home directory `/var/lib/usbmux'. 2022-03-08T22:52:28.496 INFO:teuthology.orchestra.run.smithi135.stdout:usbmuxd.service is a disabled or a static unit, not starting it. 2022-03-08T22:52:28.633 INFO:teuthology.orchestra.run.smithi135.stdout:Setting up thermald (1.9.1-1ubuntu0.6) ... 2022-03-08T22:52:28.649 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up linux-image-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-08T22:52:28.975 INFO:teuthology.orchestra.run.smithi135.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.thermald.service -> /lib/systemd/system/thermald.service. 2022-03-08T22:52:28.975 INFO:teuthology.orchestra.run.smithi135.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/thermald.service -> /lib/systemd/system/thermald.service. 2022-03-08T22:52:29.324 INFO:teuthology.orchestra.run.smithi106.stdout:I: /boot/vmlinuz is now a symlink to vmlinuz-5.13.0-35-generic 2022-03-08T22:52:29.325 INFO:teuthology.orchestra.run.smithi106.stdout:I: /boot/initrd.img is now a symlink to initrd.img-5.13.0-35-generic 2022-03-08T22:52:29.457 INFO:teuthology.orchestra.run.smithi135.stdout:Setting up linux-image-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-08T22:52:29.495 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up linux-modules-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-08T22:52:29.953 INFO:teuthology.orchestra.run.smithi077.stdout:I: /boot/vmlinuz is now a symlink to vmlinuz-5.13.0-35-generic 2022-03-08T22:52:29.954 INFO:teuthology.orchestra.run.smithi077.stdout:I: /boot/initrd.img is now a symlink to initrd.img-5.13.0-35-generic 2022-03-08T22:52:30.126 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up linux-modules-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-08T22:52:30.696 INFO:teuthology.orchestra.run.smithi135.stdout:I: /boot/vmlinuz is now a symlink to vmlinuz-5.13.0-35-generic 2022-03-08T22:52:30.696 INFO:teuthology.orchestra.run.smithi135.stdout:I: /boot/initrd.img is now a symlink to initrd.img-5.13.0-35-generic 2022-03-08T22:52:30.829 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up linux-modules-extra-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-08T22:52:30.854 INFO:teuthology.orchestra.run.smithi135.stdout:Setting up linux-modules-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-08T22:52:31.486 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up linux-modules-extra-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-08T22:52:32.106 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up linux-image-generic-hwe-20.04 (5.13.0.35.40~20.04.20) ... 2022-03-08T22:52:32.172 INFO:teuthology.orchestra.run.smithi135.stdout:Setting up linux-modules-extra-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-08T22:52:32.215 INFO:teuthology.orchestra.run.smithi106.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.2) ... 2022-03-08T22:52:32.804 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up linux-image-generic-hwe-20.04 (5.13.0.35.40~20.04.20) ... 2022-03-08T22:52:32.930 INFO:teuthology.orchestra.run.smithi077.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.2) ... 2022-03-08T22:52:33.491 INFO:teuthology.orchestra.run.smithi135.stdout:Setting up linux-image-generic-hwe-20.04 (5.13.0.35.40~20.04.20) ... 2022-03-08T22:52:33.617 INFO:teuthology.orchestra.run.smithi135.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.2) ... 2022-03-08T22:52:36.321 INFO:teuthology.orchestra.run.smithi106.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-03-08T22:52:36.685 INFO:teuthology.orchestra.run.smithi077.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-03-08T22:52:36.956 INFO:teuthology.orchestra.run.smithi106.stdout:Processing triggers for dbus (1.12.16-2ubuntu2.1) ... 2022-03-08T22:52:37.040 INFO:teuthology.orchestra.run.smithi106.stdout:Processing triggers for linux-image-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-08T22:52:37.236 INFO:teuthology.orchestra.run.smithi106.stdout:/etc/kernel/postinst.d/initramfs-tools: 2022-03-08T22:52:37.236 INFO:teuthology.orchestra.run.smithi106.stdout:update-initramfs: Generating /boot/initrd.img-5.13.0-35-generic 2022-03-08T22:52:38.138 INFO:teuthology.orchestra.run.smithi077.stdout:Processing triggers for dbus (1.12.16-2ubuntu2.1) ... 2022-03-08T22:52:38.216 INFO:teuthology.orchestra.run.smithi135.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-03-08T22:52:38.230 INFO:teuthology.orchestra.run.smithi077.stdout:Processing triggers for linux-image-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-08T22:52:38.424 INFO:teuthology.orchestra.run.smithi077.stdout:/etc/kernel/postinst.d/initramfs-tools: 2022-03-08T22:52:38.425 INFO:teuthology.orchestra.run.smithi077.stdout:update-initramfs: Generating /boot/initrd.img-5.13.0-35-generic 2022-03-08T22:52:38.910 INFO:teuthology.orchestra.run.smithi135.stdout:Processing triggers for dbus (1.12.16-2ubuntu2.1) ... 2022-03-08T22:52:39.011 INFO:teuthology.orchestra.run.smithi135.stdout:Processing triggers for linux-image-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-08T22:52:39.206 INFO:teuthology.orchestra.run.smithi135.stdout:/etc/kernel/postinst.d/initramfs-tools: 2022-03-08T22:52:39.206 INFO:teuthology.orchestra.run.smithi135.stdout:update-initramfs: Generating /boot/initrd.img-5.13.0-35-generic 2022-03-08T22:53:00.678 INFO:teuthology.orchestra.run.smithi106.stdout:/etc/kernel/postinst.d/zz-update-grub: 2022-03-08T22:53:00.678 INFO:teuthology.orchestra.run.smithi106.stdout:Sourcing file `/etc/default/grub' 2022-03-08T22:53:00.693 INFO:teuthology.orchestra.run.smithi106.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-03-08T22:53:00.714 INFO:teuthology.orchestra.run.smithi106.stdout:Generating grub configuration file ... 2022-03-08T22:53:01.013 INFO:teuthology.orchestra.run.smithi106.stdout:Found linux image: /boot/vmlinuz-5.13.0-35-generic 2022-03-08T22:53:01.047 INFO:teuthology.orchestra.run.smithi106.stdout:Found initrd image: /boot/initrd.img-5.13.0-35-generic 2022-03-08T22:53:01.418 INFO:teuthology.orchestra.run.smithi106.stdout:Found linux image: /boot/vmlinuz-5.4.0-97-generic 2022-03-08T22:53:01.431 INFO:teuthology.orchestra.run.smithi106.stdout:Found initrd image: /boot/initrd.img-5.4.0-97-generic 2022-03-08T22:53:01.932 INFO:teuthology.orchestra.run.smithi077.stdout:/etc/kernel/postinst.d/zz-update-grub: 2022-03-08T22:53:01.932 INFO:teuthology.orchestra.run.smithi077.stdout:Sourcing file `/etc/default/grub' 2022-03-08T22:53:01.944 INFO:teuthology.orchestra.run.smithi077.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-03-08T22:53:01.960 INFO:teuthology.orchestra.run.smithi077.stdout:Generating grub configuration file ... 2022-03-08T22:53:02.283 INFO:teuthology.orchestra.run.smithi077.stdout:Found linux image: /boot/vmlinuz-5.13.0-35-generic 2022-03-08T22:53:02.316 INFO:teuthology.orchestra.run.smithi077.stdout:Found initrd image: /boot/initrd.img-5.13.0-35-generic 2022-03-08T22:53:02.723 INFO:teuthology.orchestra.run.smithi077.stdout:Found linux image: /boot/vmlinuz-5.4.0-97-generic 2022-03-08T22:53:02.735 INFO:teuthology.orchestra.run.smithi077.stdout:Found initrd image: /boot/initrd.img-5.4.0-97-generic 2022-03-08T22:53:02.827 INFO:teuthology.orchestra.run.smithi135.stdout:/etc/kernel/postinst.d/zz-update-grub: 2022-03-08T22:53:02.827 INFO:teuthology.orchestra.run.smithi135.stdout:Sourcing file `/etc/default/grub' 2022-03-08T22:53:02.843 INFO:teuthology.orchestra.run.smithi135.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-03-08T22:53:02.869 INFO:teuthology.orchestra.run.smithi135.stdout:Generating grub configuration file ... 2022-03-08T22:53:03.158 INFO:teuthology.orchestra.run.smithi135.stdout:Found linux image: /boot/vmlinuz-5.13.0-35-generic 2022-03-08T22:53:03.187 INFO:teuthology.orchestra.run.smithi135.stdout:Found initrd image: /boot/initrd.img-5.13.0-35-generic 2022-03-08T22:53:03.568 INFO:teuthology.orchestra.run.smithi135.stdout:Found linux image: /boot/vmlinuz-5.4.0-97-generic 2022-03-08T22:53:03.579 INFO:teuthology.orchestra.run.smithi135.stdout:Found initrd image: /boot/initrd.img-5.4.0-97-generic 2022-03-08T22:53:04.065 INFO:teuthology.orchestra.run.smithi106.stdout:done 2022-03-08T22:53:04.425 DEBUG:teuthology.orchestra.run.smithi106:> dpkg -s linux-image-generic-hwe-20.04 2022-03-08T22:53:04.467 INFO:teuthology.orchestra.run.smithi106.stdout:Package: linux-image-generic-hwe-20.04 2022-03-08T22:53:04.467 INFO:teuthology.orchestra.run.smithi106.stdout:Status: install ok installed 2022-03-08T22:53:04.468 INFO:teuthology.orchestra.run.smithi106.stdout:Priority: optional 2022-03-08T22:53:04.468 INFO:teuthology.orchestra.run.smithi106.stdout:Section: kernel 2022-03-08T22:53:04.468 INFO:teuthology.orchestra.run.smithi106.stdout:Installed-Size: 19 2022-03-08T22:53:04.468 INFO:teuthology.orchestra.run.smithi106.stdout:Maintainer: Ubuntu Kernel Team 2022-03-08T22:53:04.469 INFO:teuthology.orchestra.run.smithi106.stdout:Architecture: amd64 2022-03-08T22:53:04.469 INFO:teuthology.orchestra.run.smithi106.stdout:Source: linux-meta-hwe-5.13 2022-03-08T22:53:04.469 INFO:teuthology.orchestra.run.smithi106.stdout:Version: 5.13.0.35.40~20.04.20 2022-03-08T22:53:04.469 INFO:teuthology.orchestra.run.smithi106.stdout:Provides: virtualbox-guest-modules (= 5.13.0-35), wireguard-modules (= 1.0.0), zfs-modules (= 2.0.6-1ubuntu2) 2022-03-08T22:53:04.469 INFO:teuthology.orchestra.run.smithi106.stdout:Depends: linux-image-5.13.0-35-generic, linux-modules-extra-5.13.0-35-generic, linux-firmware, intel-microcode, amd64-microcode 2022-03-08T22:53:04.470 INFO:teuthology.orchestra.run.smithi106.stdout:Recommends: thermald 2022-03-08T22:53:04.470 INFO:teuthology.orchestra.run.smithi106.stdout:Description: Generic Linux kernel image 2022-03-08T22:53:04.470 INFO:teuthology.orchestra.run.smithi106.stdout: This package will always depend on the latest generic kernel image 2022-03-08T22:53:04.470 INFO:teuthology.orchestra.run.smithi106.stdout: available. 2022-03-08T22:53:04.471 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-97-generic Expected: 5.13.0-35-generic 2022-03-08T22:53:04.471 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2022-03-08T22:53:04.471 DEBUG:teuthology.task.kernel:src is distro, skipping download 2022-03-08T22:53:04.472 INFO:teuthology.task.kernel:Installing distro kernel on host.b... 2022-03-08T22:53:04.472 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi106.front.sepia.ceph.com, path=None, version=distro) 2022-03-08T22:53:04.472 DEBUG:teuthology.orchestra.run.smithi106:> sudo apt-get clean 2022-03-08T22:53:04.525 DEBUG:teuthology.orchestra.run.smithi106:> sudo apt-get update 2022-03-08T22:53:04.686 INFO:teuthology.orchestra.run.smithi106.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-03-08T22:53:04.802 INFO:teuthology.orchestra.run.smithi106.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-08T22:53:04.884 INFO:teuthology.orchestra.run.smithi106.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-03-08T22:53:04.967 INFO:teuthology.orchestra.run.smithi106.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-03-08T22:53:05.343 INFO:teuthology.orchestra.run.smithi077.stdout:done 2022-03-08T22:53:05.397 INFO:teuthology.orchestra.run.smithi135.stdout:done 2022-03-08T22:53:05.676 DEBUG:teuthology.orchestra.run.smithi077:> dpkg -s linux-image-generic-hwe-20.04 2022-03-08T22:53:05.721 INFO:teuthology.orchestra.run.smithi077.stdout:Package: linux-image-generic-hwe-20.04 2022-03-08T22:53:05.721 INFO:teuthology.orchestra.run.smithi077.stdout:Status: install ok installed 2022-03-08T22:53:05.721 INFO:teuthology.orchestra.run.smithi077.stdout:Priority: optional 2022-03-08T22:53:05.722 INFO:teuthology.orchestra.run.smithi077.stdout:Section: kernel 2022-03-08T22:53:05.722 INFO:teuthology.orchestra.run.smithi077.stdout:Installed-Size: 19 2022-03-08T22:53:05.722 INFO:teuthology.orchestra.run.smithi077.stdout:Maintainer: Ubuntu Kernel Team 2022-03-08T22:53:05.722 INFO:teuthology.orchestra.run.smithi077.stdout:Architecture: amd64 2022-03-08T22:53:05.722 INFO:teuthology.orchestra.run.smithi077.stdout:Source: linux-meta-hwe-5.13 2022-03-08T22:53:05.723 INFO:teuthology.orchestra.run.smithi077.stdout:Version: 5.13.0.35.40~20.04.20 2022-03-08T22:53:05.723 INFO:teuthology.orchestra.run.smithi077.stdout:Provides: virtualbox-guest-modules (= 5.13.0-35), wireguard-modules (= 1.0.0), zfs-modules (= 2.0.6-1ubuntu2) 2022-03-08T22:53:05.723 INFO:teuthology.orchestra.run.smithi077.stdout:Depends: linux-image-5.13.0-35-generic, linux-modules-extra-5.13.0-35-generic, linux-firmware, intel-microcode, amd64-microcode 2022-03-08T22:53:05.723 INFO:teuthology.orchestra.run.smithi077.stdout:Recommends: thermald 2022-03-08T22:53:05.724 INFO:teuthology.orchestra.run.smithi077.stdout:Description: Generic Linux kernel image 2022-03-08T22:53:05.724 INFO:teuthology.orchestra.run.smithi077.stdout: This package will always depend on the latest generic kernel image 2022-03-08T22:53:05.724 INFO:teuthology.orchestra.run.smithi077.stdout: available. 2022-03-08T22:53:05.724 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-97-generic Expected: 5.13.0-35-generic 2022-03-08T22:53:05.725 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2022-03-08T22:53:05.725 DEBUG:teuthology.task.kernel:src is distro, skipping download 2022-03-08T22:53:05.725 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2022-03-08T22:53:05.725 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi077.front.sepia.ceph.com, path=None, version=distro) 2022-03-08T22:53:05.726 DEBUG:teuthology.orchestra.run.smithi077:> sudo apt-get clean 2022-03-08T22:53:05.771 DEBUG:teuthology.orchestra.run.smithi135:> dpkg -s linux-image-generic-hwe-20.04 2022-03-08T22:53:05.792 DEBUG:teuthology.orchestra.run.smithi077:> sudo apt-get update 2022-03-08T22:53:05.804 INFO:teuthology.orchestra.run.smithi135.stdout:Package: linux-image-generic-hwe-20.04 2022-03-08T22:53:05.804 INFO:teuthology.orchestra.run.smithi135.stdout:Status: install ok installed 2022-03-08T22:53:05.804 INFO:teuthology.orchestra.run.smithi135.stdout:Priority: optional 2022-03-08T22:53:05.805 INFO:teuthology.orchestra.run.smithi135.stdout:Section: kernel 2022-03-08T22:53:05.805 INFO:teuthology.orchestra.run.smithi135.stdout:Installed-Size: 19 2022-03-08T22:53:05.805 INFO:teuthology.orchestra.run.smithi135.stdout:Maintainer: Ubuntu Kernel Team 2022-03-08T22:53:05.805 INFO:teuthology.orchestra.run.smithi135.stdout:Architecture: amd64 2022-03-08T22:53:05.805 INFO:teuthology.orchestra.run.smithi135.stdout:Source: linux-meta-hwe-5.13 2022-03-08T22:53:05.806 INFO:teuthology.orchestra.run.smithi135.stdout:Version: 5.13.0.35.40~20.04.20 2022-03-08T22:53:05.806 INFO:teuthology.orchestra.run.smithi135.stdout:Provides: virtualbox-guest-modules (= 5.13.0-35), wireguard-modules (= 1.0.0), zfs-modules (= 2.0.6-1ubuntu2) 2022-03-08T22:53:05.806 INFO:teuthology.orchestra.run.smithi135.stdout:Depends: linux-image-5.13.0-35-generic, linux-modules-extra-5.13.0-35-generic, linux-firmware, intel-microcode, amd64-microcode 2022-03-08T22:53:05.806 INFO:teuthology.orchestra.run.smithi135.stdout:Recommends: thermald 2022-03-08T22:53:05.807 INFO:teuthology.orchestra.run.smithi135.stdout:Description: Generic Linux kernel image 2022-03-08T22:53:05.807 INFO:teuthology.orchestra.run.smithi135.stdout: This package will always depend on the latest generic kernel image 2022-03-08T22:53:05.807 INFO:teuthology.orchestra.run.smithi135.stdout: available. 2022-03-08T22:53:05.807 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-97-generic Expected: 5.13.0-35-generic 2022-03-08T22:53:05.808 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2022-03-08T22:53:05.808 DEBUG:teuthology.task.kernel:src is distro, skipping download 2022-03-08T22:53:05.808 INFO:teuthology.task.kernel:Installing distro kernel on host.c... 2022-03-08T22:53:05.808 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi135.front.sepia.ceph.com, path=None, version=distro) 2022-03-08T22:53:05.809 DEBUG:teuthology.orchestra.run.smithi135:> sudo apt-get clean 2022-03-08T22:53:05.884 DEBUG:teuthology.orchestra.run.smithi135:> sudo apt-get update 2022-03-08T22:53:05.928 INFO:teuthology.orchestra.run.smithi077.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-03-08T22:53:06.020 INFO:teuthology.orchestra.run.smithi135.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-03-08T22:53:06.042 INFO:teuthology.orchestra.run.smithi077.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-08T22:53:06.125 INFO:teuthology.orchestra.run.smithi077.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-03-08T22:53:06.134 INFO:teuthology.orchestra.run.smithi135.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-08T22:53:06.211 INFO:teuthology.orchestra.run.smithi077.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-03-08T22:53:06.217 INFO:teuthology.orchestra.run.smithi135.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-03-08T22:53:06.218 INFO:teuthology.orchestra.run.smithi106.stdout:Reading package lists... 2022-03-08T22:53:06.242 DEBUG:teuthology.orchestra.run.smithi106:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic-hwe-20.04 2022-03-08T22:53:06.299 INFO:teuthology.orchestra.run.smithi135.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-03-08T22:53:06.307 INFO:teuthology.orchestra.run.smithi106.stdout:Reading package lists... 2022-03-08T22:53:06.479 INFO:teuthology.orchestra.run.smithi106.stdout:Building dependency tree... 2022-03-08T22:53:06.480 INFO:teuthology.orchestra.run.smithi106.stdout:Reading state information... 2022-03-08T22:53:06.660 INFO:teuthology.orchestra.run.smithi106.stdout:linux-image-generic-hwe-20.04 is already the newest version (5.13.0.35.40~20.04.20). 2022-03-08T22:53:06.661 INFO:teuthology.orchestra.run.smithi106.stdout:0 upgraded, 0 newly installed, 0 to remove and 92 not upgraded. 2022-03-08T22:53:06.662 DEBUG:teuthology.orchestra.run.smithi106:> dpkg -s linux-image-generic-hwe-20.04 2022-03-08T22:53:06.682 INFO:teuthology.orchestra.run.smithi106.stdout:Package: linux-image-generic-hwe-20.04 2022-03-08T22:53:06.683 INFO:teuthology.orchestra.run.smithi106.stdout:Status: install ok installed 2022-03-08T22:53:06.683 INFO:teuthology.orchestra.run.smithi106.stdout:Priority: optional 2022-03-08T22:53:06.683 INFO:teuthology.orchestra.run.smithi106.stdout:Section: kernel 2022-03-08T22:53:06.683 INFO:teuthology.orchestra.run.smithi106.stdout:Installed-Size: 19 2022-03-08T22:53:06.683 INFO:teuthology.orchestra.run.smithi106.stdout:Maintainer: Ubuntu Kernel Team 2022-03-08T22:53:06.684 INFO:teuthology.orchestra.run.smithi106.stdout:Architecture: amd64 2022-03-08T22:53:06.684 INFO:teuthology.orchestra.run.smithi106.stdout:Source: linux-meta-hwe-5.13 2022-03-08T22:53:06.684 INFO:teuthology.orchestra.run.smithi106.stdout:Version: 5.13.0.35.40~20.04.20 2022-03-08T22:53:06.684 INFO:teuthology.orchestra.run.smithi106.stdout:Provides: virtualbox-guest-modules (= 5.13.0-35), wireguard-modules (= 1.0.0), zfs-modules (= 2.0.6-1ubuntu2) 2022-03-08T22:53:06.684 INFO:teuthology.orchestra.run.smithi106.stdout:Depends: linux-image-5.13.0-35-generic, linux-modules-extra-5.13.0-35-generic, linux-firmware, intel-microcode, amd64-microcode 2022-03-08T22:53:06.685 INFO:teuthology.orchestra.run.smithi106.stdout:Recommends: thermald 2022-03-08T22:53:06.685 INFO:teuthology.orchestra.run.smithi106.stdout:Description: Generic Linux kernel image 2022-03-08T22:53:06.685 INFO:teuthology.orchestra.run.smithi106.stdout: This package will always depend on the latest generic kernel image 2022-03-08T22:53:06.685 INFO:teuthology.orchestra.run.smithi106.stdout: available. 2022-03-08T22:53:06.686 DEBUG:teuthology.orchestra.run.smithi106:> mktemp 2022-03-08T22:53:06.691 INFO:teuthology.orchestra.run.smithi106.stdout:/tmp/tmp.7S1e4elZPJ 2022-03-08T22:53:06.691 DEBUG:teuthology.orchestra.run.smithi106:> sudo cp /boot/grub/grub.cfg /tmp/tmp.7S1e4elZPJ 2022-03-08T22:53:06.750 DEBUG:teuthology.orchestra.run.smithi106:> sudo chmod 0666 /tmp/tmp.7S1e4elZPJ 2022-03-08T22:53:06.788 DEBUG:teuthology.orchestra.remote:smithi106:/tmp/tmp.7S1e4elZPJ is 8KB 2022-03-08T22:53:06.803 DEBUG:teuthology.orchestra.run.smithi106:> rm -fr /tmp/tmp.7S1e4elZPJ 2022-03-08T22:53:06.809 DEBUG:teuthology.orchestra.run.smithi106:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2022-03-08T22:53:06.865 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2022-03-08T22:53:06.866 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd of=/etc/grub.d/01_ceph_kernel 2022-03-08T22:53:06.866 DEBUG:teuthology.orchestra.run.smithi106:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2022-03-08T22:53:06.898 INFO:teuthology.task.kernel:Distro Kernel Version: 5.13.0-35-generic 2022-03-08T22:53:06.899 DEBUG:teuthology.orchestra.run.smithi106:> sudo update-grub 2022-03-08T22:53:07.456 INFO:teuthology.orchestra.run.smithi077.stdout:Reading package lists... 2022-03-08T22:53:07.478 DEBUG:teuthology.orchestra.run.smithi077:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic-hwe-20.04 2022-03-08T22:53:07.544 INFO:teuthology.orchestra.run.smithi077.stdout:Reading package lists... 2022-03-08T22:53:07.545 INFO:teuthology.orchestra.run.smithi135.stdout:Reading package lists... 2022-03-08T22:53:07.567 DEBUG:teuthology.orchestra.run.smithi135:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic-hwe-20.04 2022-03-08T22:53:07.636 INFO:teuthology.orchestra.run.smithi135.stdout:Reading package lists... 2022-03-08T22:53:07.697 INFO:teuthology.orchestra.run.smithi077.stdout:Building dependency tree... 2022-03-08T22:53:07.698 INFO:teuthology.orchestra.run.smithi077.stdout:Reading state information... 2022-03-08T22:53:07.789 INFO:teuthology.orchestra.run.smithi135.stdout:Building dependency tree... 2022-03-08T22:53:07.790 INFO:teuthology.orchestra.run.smithi135.stdout:Reading state information... 2022-03-08T22:53:07.865 INFO:teuthology.orchestra.run.smithi077.stdout:linux-image-generic-hwe-20.04 is already the newest version (5.13.0.35.40~20.04.20). 2022-03-08T22:53:07.865 INFO:teuthology.orchestra.run.smithi077.stdout:0 upgraded, 0 newly installed, 0 to remove and 92 not upgraded. 2022-03-08T22:53:07.867 DEBUG:teuthology.orchestra.run.smithi077:> dpkg -s linux-image-generic-hwe-20.04 2022-03-08T22:53:07.887 INFO:teuthology.orchestra.run.smithi077.stdout:Package: linux-image-generic-hwe-20.04 2022-03-08T22:53:07.888 INFO:teuthology.orchestra.run.smithi077.stdout:Status: install ok installed 2022-03-08T22:53:07.888 INFO:teuthology.orchestra.run.smithi077.stdout:Priority: optional 2022-03-08T22:53:07.888 INFO:teuthology.orchestra.run.smithi077.stdout:Section: kernel 2022-03-08T22:53:07.889 INFO:teuthology.orchestra.run.smithi077.stdout:Installed-Size: 19 2022-03-08T22:53:07.889 INFO:teuthology.orchestra.run.smithi077.stdout:Maintainer: Ubuntu Kernel Team 2022-03-08T22:53:07.889 INFO:teuthology.orchestra.run.smithi077.stdout:Architecture: amd64 2022-03-08T22:53:07.889 INFO:teuthology.orchestra.run.smithi077.stdout:Source: linux-meta-hwe-5.13 2022-03-08T22:53:07.890 INFO:teuthology.orchestra.run.smithi077.stdout:Version: 5.13.0.35.40~20.04.20 2022-03-08T22:53:07.890 INFO:teuthology.orchestra.run.smithi077.stdout:Provides: virtualbox-guest-modules (= 5.13.0-35), wireguard-modules (= 1.0.0), zfs-modules (= 2.0.6-1ubuntu2) 2022-03-08T22:53:07.890 INFO:teuthology.orchestra.run.smithi077.stdout:Depends: linux-image-5.13.0-35-generic, linux-modules-extra-5.13.0-35-generic, linux-firmware, intel-microcode, amd64-microcode 2022-03-08T22:53:07.890 INFO:teuthology.orchestra.run.smithi077.stdout:Recommends: thermald 2022-03-08T22:53:07.890 INFO:teuthology.orchestra.run.smithi077.stdout:Description: Generic Linux kernel image 2022-03-08T22:53:07.891 INFO:teuthology.orchestra.run.smithi077.stdout: This package will always depend on the latest generic kernel image 2022-03-08T22:53:07.891 INFO:teuthology.orchestra.run.smithi077.stdout: available. 2022-03-08T22:53:07.891 DEBUG:teuthology.orchestra.run.smithi077:> mktemp 2022-03-08T22:53:07.937 INFO:teuthology.orchestra.run.smithi077.stdout:/tmp/tmp.CJZPor92Ks 2022-03-08T22:53:07.938 DEBUG:teuthology.orchestra.run.smithi077:> sudo cp /boot/grub/grub.cfg /tmp/tmp.CJZPor92Ks 2022-03-08T22:53:07.954 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod 0666 /tmp/tmp.CJZPor92Ks 2022-03-08T22:53:07.956 INFO:teuthology.orchestra.run.smithi135.stdout:linux-image-generic-hwe-20.04 is already the newest version (5.13.0.35.40~20.04.20). 2022-03-08T22:53:07.957 INFO:teuthology.orchestra.run.smithi135.stdout:0 upgraded, 0 newly installed, 0 to remove and 92 not upgraded. 2022-03-08T22:53:07.959 DEBUG:teuthology.orchestra.run.smithi135:> dpkg -s linux-image-generic-hwe-20.04 2022-03-08T22:53:07.979 INFO:teuthology.orchestra.run.smithi135.stdout:Package: linux-image-generic-hwe-20.04 2022-03-08T22:53:07.980 INFO:teuthology.orchestra.run.smithi135.stdout:Status: install ok installed 2022-03-08T22:53:07.980 INFO:teuthology.orchestra.run.smithi135.stdout:Priority: optional 2022-03-08T22:53:07.980 INFO:teuthology.orchestra.run.smithi135.stdout:Section: kernel 2022-03-08T22:53:07.980 INFO:teuthology.orchestra.run.smithi135.stdout:Installed-Size: 19 2022-03-08T22:53:07.981 INFO:teuthology.orchestra.run.smithi135.stdout:Maintainer: Ubuntu Kernel Team 2022-03-08T22:53:07.981 INFO:teuthology.orchestra.run.smithi135.stdout:Architecture: amd64 2022-03-08T22:53:07.981 INFO:teuthology.orchestra.run.smithi135.stdout:Source: linux-meta-hwe-5.13 2022-03-08T22:53:07.981 INFO:teuthology.orchestra.run.smithi135.stdout:Version: 5.13.0.35.40~20.04.20 2022-03-08T22:53:07.982 INFO:teuthology.orchestra.run.smithi135.stdout:Provides: virtualbox-guest-modules (= 5.13.0-35), wireguard-modules (= 1.0.0), zfs-modules (= 2.0.6-1ubuntu2) 2022-03-08T22:53:07.982 INFO:teuthology.orchestra.run.smithi135.stdout:Depends: linux-image-5.13.0-35-generic, linux-modules-extra-5.13.0-35-generic, linux-firmware, intel-microcode, amd64-microcode 2022-03-08T22:53:07.982 INFO:teuthology.orchestra.run.smithi135.stdout:Recommends: thermald 2022-03-08T22:53:07.982 INFO:teuthology.orchestra.run.smithi135.stdout:Description: Generic Linux kernel image 2022-03-08T22:53:07.983 INFO:teuthology.orchestra.run.smithi135.stdout: This package will always depend on the latest generic kernel image 2022-03-08T22:53:07.983 INFO:teuthology.orchestra.run.smithi135.stdout: available. 2022-03-08T22:53:07.983 DEBUG:teuthology.orchestra.run.smithi135:> mktemp 2022-03-08T22:53:07.989 INFO:teuthology.orchestra.run.smithi135.stdout:/tmp/tmp.wSOOoZnM3K 2022-03-08T22:53:07.989 DEBUG:teuthology.orchestra.run.smithi135:> sudo cp /boot/grub/grub.cfg /tmp/tmp.wSOOoZnM3K 2022-03-08T22:53:07.997 DEBUG:teuthology.orchestra.remote:smithi077:/tmp/tmp.CJZPor92Ks is 8KB 2022-03-08T22:53:08.011 DEBUG:teuthology.orchestra.run.smithi077:> rm -fr /tmp/tmp.CJZPor92Ks 2022-03-08T22:53:08.019 DEBUG:teuthology.orchestra.run.smithi077:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2022-03-08T22:53:08.051 DEBUG:teuthology.orchestra.run.smithi135:> sudo chmod 0666 /tmp/tmp.wSOOoZnM3K 2022-03-08T22:53:08.067 INFO:teuthology.orchestra.run.smithi106.stderr:Sourcing file `/etc/default/grub' 2022-03-08T22:53:08.074 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2022-03-08T22:53:08.075 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/etc/grub.d/01_ceph_kernel 2022-03-08T22:53:08.075 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2022-03-08T22:53:08.076 INFO:teuthology.orchestra.run.smithi106.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-03-08T22:53:08.095 DEBUG:teuthology.orchestra.remote:smithi135:/tmp/tmp.wSOOoZnM3K is 8KB 2022-03-08T22:53:08.097 INFO:teuthology.orchestra.run.smithi106.stderr:Generating grub configuration file ... 2022-03-08T22:53:08.109 INFO:teuthology.task.kernel:Distro Kernel Version: 5.13.0-35-generic 2022-03-08T22:53:08.110 DEBUG:teuthology.orchestra.run.smithi077:> sudo update-grub 2022-03-08T22:53:08.111 DEBUG:teuthology.orchestra.run.smithi135:> rm -fr /tmp/tmp.wSOOoZnM3K 2022-03-08T22:53:08.118 DEBUG:teuthology.orchestra.run.smithi135:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2022-03-08T22:53:08.174 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2022-03-08T22:53:08.175 DEBUG:teuthology.orchestra.run.smithi135:> sudo dd of=/etc/grub.d/01_ceph_kernel 2022-03-08T22:53:08.175 DEBUG:teuthology.orchestra.run.smithi135:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2022-03-08T22:53:08.208 INFO:teuthology.task.kernel:Distro Kernel Version: 5.13.0-35-generic 2022-03-08T22:53:08.208 DEBUG:teuthology.orchestra.run.smithi135:> sudo update-grub 2022-03-08T22:53:08.330 INFO:teuthology.orchestra.run.smithi106.stderr:Found linux image: /boot/vmlinuz-5.13.0-35-generic 2022-03-08T22:53:08.342 INFO:teuthology.orchestra.run.smithi106.stderr:Found initrd image: /boot/initrd.img-5.13.0-35-generic 2022-03-08T22:53:08.698 INFO:teuthology.orchestra.run.smithi106.stderr:Found linux image: /boot/vmlinuz-5.4.0-97-generic 2022-03-08T22:53:08.709 INFO:teuthology.orchestra.run.smithi106.stderr:Found initrd image: /boot/initrd.img-5.4.0-97-generic 2022-03-08T22:53:09.220 INFO:teuthology.orchestra.run.smithi077.stderr:Sourcing file `/etc/default/grub' 2022-03-08T22:53:09.229 INFO:teuthology.orchestra.run.smithi077.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-03-08T22:53:09.250 INFO:teuthology.orchestra.run.smithi077.stderr:Generating grub configuration file ... 2022-03-08T22:53:09.425 INFO:teuthology.orchestra.run.smithi135.stderr:Sourcing file `/etc/default/grub' 2022-03-08T22:53:09.429 INFO:teuthology.orchestra.run.smithi135.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-03-08T22:53:09.442 INFO:teuthology.orchestra.run.smithi135.stderr:Generating grub configuration file ... 2022-03-08T22:53:09.496 INFO:teuthology.orchestra.run.smithi077.stderr:Found linux image: /boot/vmlinuz-5.13.0-35-generic 2022-03-08T22:53:09.508 INFO:teuthology.orchestra.run.smithi077.stderr:Found initrd image: /boot/initrd.img-5.13.0-35-generic 2022-03-08T22:53:09.724 INFO:teuthology.orchestra.run.smithi135.stderr:Found linux image: /boot/vmlinuz-5.13.0-35-generic 2022-03-08T22:53:09.735 INFO:teuthology.orchestra.run.smithi135.stderr:Found initrd image: /boot/initrd.img-5.13.0-35-generic 2022-03-08T22:53:09.843 INFO:teuthology.orchestra.run.smithi077.stderr:Found linux image: /boot/vmlinuz-5.4.0-97-generic 2022-03-08T22:53:09.854 INFO:teuthology.orchestra.run.smithi077.stderr:Found initrd image: /boot/initrd.img-5.4.0-97-generic 2022-03-08T22:53:10.087 INFO:teuthology.orchestra.run.smithi135.stderr:Found linux image: /boot/vmlinuz-5.4.0-97-generic 2022-03-08T22:53:10.099 INFO:teuthology.orchestra.run.smithi135.stderr:Found initrd image: /boot/initrd.img-5.4.0-97-generic 2022-03-08T22:53:10.403 INFO:teuthology.orchestra.run.smithi106.stderr:done 2022-03-08T22:53:10.404 DEBUG:teuthology.orchestra.run.smithi106:> sudo shutdown -r now 2022-03-08T22:53:10.820 INFO:teuthology.orchestra.run.smithi135.stderr:done 2022-03-08T22:53:10.821 DEBUG:teuthology.orchestra.run.smithi135:> sudo shutdown -r now 2022-03-08T22:53:11.442 INFO:teuthology.orchestra.run.smithi077.stderr:done 2022-03-08T22:53:11.444 DEBUG:teuthology.orchestra.run.smithi077:> sudo shutdown -r now 2022-03-08T22:53:40.433 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2022-03-08T22:53:40.434 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi106.front.sepia.ceph.com' 2022-03-08T22:53:40.435 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi106.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-08T22:53:40.824 INFO:teuthology.task.kernel:Checking client host.c for new kernel version... 2022-03-08T22:53:40.825 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi135.front.sepia.ceph.com' 2022-03-08T22:53:40.826 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi135.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-08T22:53:41.448 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2022-03-08T22:53:41.449 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi077.front.sepia.ceph.com' 2022-03-08T22:53:41.450 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-08T22:53:58.859 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.106 2022-03-08T22:53:59.370 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.135 2022-03-08T22:53:59.887 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.77 2022-03-08T22:54:28.882 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi106.front.sepia.ceph.com' 2022-03-08T22:54:28.883 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi106.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-08T22:54:29.372 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi135.front.sepia.ceph.com' 2022-03-08T22:54:29.373 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi135.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-08T22:54:29.889 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi077.front.sepia.ceph.com' 2022-03-08T22:54:29.890 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-08T22:54:47.243 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.106 2022-03-08T22:54:47.722 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.135 2022-03-08T22:54:48.267 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.77 2022-03-08T22:55:17.271 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi106.front.sepia.ceph.com' 2022-03-08T22:55:17.272 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi106.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-08T22:55:17.377 DEBUG:teuthology.orchestra.run.smithi106:> true 2022-03-08T22:55:17.723 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi135.front.sepia.ceph.com' 2022-03-08T22:55:17.811 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi135.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-08T22:55:17.861 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi106.front.sepia.ceph.com' 2022-03-08T22:55:17.861 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "5.13.0-35-generic"... 2022-03-08T22:55:17.862 DEBUG:teuthology.orchestra.run.smithi106:> uname -r 2022-03-08T22:55:17.869 INFO:teuthology.orchestra.run.smithi106.stdout:5.13.0-35-generic 2022-03-08T22:55:17.869 DEBUG:teuthology.task.kernel:current kernel version is 5.13.0-35-generic vs 5.13.0-35-generic 2022-03-08T22:55:17.869 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2022-03-08T22:55:17.870 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2022-03-08T22:55:17.939 DEBUG:teuthology.orchestra.run.smithi135:> true 2022-03-08T22:55:18.268 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi077.front.sepia.ceph.com' 2022-03-08T22:55:18.268 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-08T22:55:18.434 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi135.front.sepia.ceph.com' 2022-03-08T22:55:18.435 INFO:teuthology.task.kernel:Checking kernel version of host.c, want "5.13.0-35-generic"... 2022-03-08T22:55:18.435 DEBUG:teuthology.orchestra.run.smithi135:> uname -r 2022-03-08T22:55:18.440 INFO:teuthology.orchestra.run.smithi135.stdout:5.13.0-35-generic 2022-03-08T22:55:18.440 DEBUG:teuthology.task.kernel:current kernel version is 5.13.0-35-generic vs 5.13.0-35-generic 2022-03-08T22:55:18.441 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2022-03-08T22:55:18.441 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2022-03-08T22:55:18.449 DEBUG:teuthology.orchestra.run.smithi077:> true 2022-03-08T22:55:18.870 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2022-03-08T22:55:18.871 DEBUG:teuthology.orchestra.run.smithi106:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2022-03-08T22:55:18.927 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi077.front.sepia.ceph.com' 2022-03-08T22:55:18.928 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "5.13.0-35-generic"... 2022-03-08T22:55:18.928 DEBUG:teuthology.orchestra.run.smithi077:> uname -r 2022-03-08T22:55:18.933 INFO:teuthology.orchestra.run.smithi077.stdout:5.13.0-35-generic 2022-03-08T22:55:18.933 DEBUG:teuthology.task.kernel:current kernel version is 5.13.0-35-generic vs 5.13.0-35-generic 2022-03-08T22:55:18.934 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2022-03-08T22:55:18.934 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2022-03-08T22:55:18.937 INFO:teuthology.orchestra.run.smithi106.stdout:ttyS1 2022-03-08T22:55:18.969 DEBUG:teuthology.parallel:result is None 2022-03-08T22:55:19.442 INFO:teuthology.task.kernel:Enabling kdb on host.c... 2022-03-08T22:55:19.442 DEBUG:teuthology.orchestra.run.smithi135:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2022-03-08T22:55:19.505 INFO:teuthology.orchestra.run.smithi135.stdout:ttyS1 2022-03-08T22:55:19.524 DEBUG:teuthology.parallel:result is None 2022-03-08T22:55:19.934 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2022-03-08T22:55:19.935 DEBUG:teuthology.orchestra.run.smithi077:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2022-03-08T22:55:20.008 INFO:teuthology.orchestra.run.smithi077.stdout:ttyS1 2022-03-08T22:55:20.030 DEBUG:teuthology.parallel:result is None 2022-03-08T22:55:20.031 INFO:teuthology.run_tasks:Running task internal.base... 2022-03-08T22:55:20.040 INFO:teuthology.task.internal:Creating test directory... 2022-03-08T22:55:20.041 DEBUG:teuthology.orchestra.run.smithi077:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2022-03-08T22:55:20.043 DEBUG:teuthology.orchestra.run.smithi106:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2022-03-08T22:55:20.046 DEBUG:teuthology.orchestra.run.smithi135:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2022-03-08T22:55:20.053 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2022-03-08T22:55:20.197 INFO:teuthology.run_tasks:Running task internal.archive... 2022-03-08T22:55:20.235 INFO:teuthology.task.internal:Creating archive directory... 2022-03-08T22:55:20.235 DEBUG:teuthology.orchestra.run.smithi077:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2022-03-08T22:55:20.238 DEBUG:teuthology.orchestra.run.smithi106:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2022-03-08T22:55:20.241 DEBUG:teuthology.orchestra.run.smithi135:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2022-03-08T22:55:20.266 INFO:teuthology.run_tasks:Running task internal.coredump... 2022-03-08T22:55:20.275 INFO:teuthology.task.internal:Enabling coredump saving... 2022-03-08T22:55:20.276 DEBUG:teuthology.orchestra.run.smithi077:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2022-03-08T22:55:20.278 DEBUG:teuthology.orchestra.run.smithi106:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2022-03-08T22:55:20.281 DEBUG:teuthology.orchestra.run.smithi135:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2022-03-08T22:55:20.301 INFO:teuthology.orchestra.run.smithi077.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-03-08T22:55:20.302 INFO:teuthology.orchestra.run.smithi106.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-03-08T22:55:20.312 INFO:teuthology.orchestra.run.smithi077.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-03-08T22:55:20.312 INFO:teuthology.orchestra.run.smithi106.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-03-08T22:55:20.324 INFO:teuthology.orchestra.run.smithi135.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-03-08T22:55:20.332 INFO:teuthology.orchestra.run.smithi135.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-03-08T22:55:20.333 INFO:teuthology.run_tasks:Running task internal.sudo... 2022-03-08T22:55:20.343 INFO:teuthology.task.internal:Configuring sudo... 2022-03-08T22:55:20.344 DEBUG:teuthology.orchestra.run.smithi077:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2022-03-08T22:55:20.346 DEBUG:teuthology.orchestra.run.smithi106:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2022-03-08T22:55:20.349 DEBUG:teuthology.orchestra.run.smithi135:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2022-03-08T22:55:20.366 INFO:teuthology.run_tasks:Running task internal.syslog... 2022-03-08T22:55:20.378 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2022-03-08T22:55:20.378 DEBUG:teuthology.orchestra.run.smithi077:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2022-03-08T22:55:20.406 DEBUG:teuthology.orchestra.run.smithi106:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2022-03-08T22:55:20.408 DEBUG:teuthology.orchestra.run.smithi135:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2022-03-08T22:55:20.416 DEBUG:teuthology.orchestra.run.smithi077:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2022-03-08T22:55:20.422 DEBUG:teuthology.orchestra.run.smithi077:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2022-03-08T22:55:20.427 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2022-03-08T22:55:20.428 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2022-03-08T22:55:20.450 DEBUG:teuthology.orchestra.run.smithi106:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2022-03-08T22:55:20.455 DEBUG:teuthology.orchestra.run.smithi106:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2022-03-08T22:55:20.463 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2022-03-08T22:55:20.463 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2022-03-08T22:55:20.489 DEBUG:teuthology.orchestra.run.smithi135:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2022-03-08T22:55:20.497 DEBUG:teuthology.orchestra.run.smithi135:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2022-03-08T22:55:20.505 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2022-03-08T22:55:20.505 DEBUG:teuthology.orchestra.run.smithi135:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2022-03-08T22:55:20.527 DEBUG:teuthology.orchestra.run.smithi077:> sudo service rsyslog restart 2022-03-08T22:55:20.530 DEBUG:teuthology.orchestra.run.smithi106:> sudo service rsyslog restart 2022-03-08T22:55:20.533 DEBUG:teuthology.orchestra.run.smithi135:> sudo service rsyslog restart 2022-03-08T22:55:20.611 INFO:teuthology.run_tasks:Running task internal.timer... 2022-03-08T22:55:20.625 INFO:teuthology.task.internal:Starting timer... 2022-03-08T22:55:20.626 INFO:teuthology.run_tasks:Running task pcp... 2022-03-08T22:55:20.671 INFO:teuthology.run_tasks:Running task selinux... 2022-03-08T22:55:20.704 DEBUG:teuthology.task.selinux:Excluding smithi077: OS 'ubuntu' does not support SELinux 2022-03-08T22:55:20.705 DEBUG:teuthology.task.selinux:Excluding smithi106: OS 'ubuntu' does not support SELinux 2022-03-08T22:55:20.705 DEBUG:teuthology.task.selinux:Excluding smithi135: OS 'ubuntu' does not support SELinux 2022-03-08T22:55:20.705 DEBUG:teuthology.task.selinux:Getting current SELinux state 2022-03-08T22:55:20.706 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2022-03-08T22:55:20.706 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2022-03-08T22:55:20.706 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2022-03-08T22:55:20.744 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master was just updated or references a specific commit; assuming it is current 2022-03-08T22:55:20.745 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master to origin/master 2022-03-08T22:55:20.766 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2022-03-08T22:55:20.768 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi077.front.sepia.ceph.com,smithi106.front.sepia.ceph.com,smithi135.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/cephlab.yml 2022-03-08T22:58:27.966 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi077.front.sepia.ceph.com'), Remote(name='ubuntu@smithi106.front.sepia.ceph.com'), Remote(name='ubuntu@smithi135.front.sepia.ceph.com')] 2022-03-08T22:58:27.968 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi077.front.sepia.ceph.com' 2022-03-08T22:58:27.970 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-08T22:58:28.013 DEBUG:teuthology.orchestra.run.smithi077:> true 2022-03-08T22:58:28.133 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi106.front.sepia.ceph.com' 2022-03-08T22:58:28.133 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi106.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-08T22:58:28.191 DEBUG:teuthology.orchestra.run.smithi106:> true 2022-03-08T22:58:28.308 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi135.front.sepia.ceph.com' 2022-03-08T22:58:28.309 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi135.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-08T22:58:28.355 DEBUG:teuthology.orchestra.run.smithi135:> true 2022-03-08T22:58:28.472 INFO:teuthology.run_tasks:Running task clock... 2022-03-08T22:58:28.483 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2022-03-08T22:58:28.484 INFO:teuthology.orchestra.run:Running command with timeout 360 2022-03-08T22:58:28.486 DEBUG:teuthology.orchestra.run.smithi077:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-03-08T22:58:28.488 INFO:teuthology.orchestra.run:Running command with timeout 360 2022-03-08T22:58:28.489 DEBUG:teuthology.orchestra.run.smithi106:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-03-08T22:58:28.491 INFO:teuthology.orchestra.run:Running command with timeout 360 2022-03-08T22:58:28.492 DEBUG:teuthology.orchestra.run.smithi135:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-03-08T22:58:28.520 INFO:teuthology.orchestra.run.smithi106.stdout: 8 Mar 22:58:28 ntpd[9027]: ntpd 4.2.8p12@1.3728-o (1): Starting 2022-03-08T22:58:28.520 INFO:teuthology.orchestra.run.smithi106.stdout: 8 Mar 22:58:28 ntpd[9027]: Command line: ntpd -gq 2022-03-08T22:58:28.521 INFO:teuthology.orchestra.run.smithi077.stdout: 8 Mar 22:58:28 ntpd[9066]: ntpd 4.2.8p12@1.3728-o (1): Starting 2022-03-08T22:58:28.522 INFO:teuthology.orchestra.run.smithi077.stdout: 8 Mar 22:58:28 ntpd[9066]: Command line: ntpd -gq 2022-03-08T22:58:28.522 INFO:teuthology.orchestra.run.smithi106.stdout: 8 Mar 22:58:28 ntpd[9027]: proto: precision = 0.078 usec (-24) 2022-03-08T22:58:28.522 INFO:teuthology.orchestra.run.smithi106.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-03-08T22:58:28.523 INFO:teuthology.orchestra.run.smithi106.stdout: 8 Mar 22:58:28 ntpd[9027]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-03-08T22:58:28.523 INFO:teuthology.orchestra.run.smithi106.stderr:restrict ::: KOD does nothing without LIMITED. 2022-03-08T22:58:28.525 INFO:teuthology.orchestra.run.smithi106.stdout: 8 Mar 22:58:28 ntpd[9027]: restrict ::: KOD does nothing without LIMITED. 2022-03-08T22:58:28.525 INFO:teuthology.orchestra.run.smithi106.stdout: 8 Mar 22:58:28 ntpd[9027]: Listen and drop on 0 v6wildcard [::]:123 2022-03-08T22:58:28.525 INFO:teuthology.orchestra.run.smithi106.stdout: 8 Mar 22:58:28 ntpd[9027]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2022-03-08T22:58:28.525 INFO:teuthology.orchestra.run.smithi106.stdout: 8 Mar 22:58:28 ntpd[9027]: Listen normally on 2 lo 127.0.0.1:123 2022-03-08T22:58:28.526 INFO:teuthology.orchestra.run.smithi106.stdout: 8 Mar 22:58:28 ntpd[9027]: Listen normally on 3 enp3s0f1 172.21.15.106:123 2022-03-08T22:58:28.526 INFO:teuthology.orchestra.run.smithi106.stdout: 8 Mar 22:58:28 ntpd[9027]: Listen normally on 4 lo [::1]:123 2022-03-08T22:58:28.526 INFO:teuthology.orchestra.run.smithi106.stdout: 8 Mar 22:58:28 ntpd[9027]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:ce6d%5]:123 2022-03-08T22:58:28.527 INFO:teuthology.orchestra.run.smithi106.stdout: 8 Mar 22:58:28 ntpd[9027]: Listening on routing socket on fd #22 for interface updates 2022-03-08T22:58:28.527 INFO:teuthology.orchestra.run.smithi077.stdout: 8 Mar 22:58:28 ntpd[9066]: proto: precision = 0.092 usec (-23) 2022-03-08T22:58:28.528 INFO:teuthology.orchestra.run.smithi135.stdout: 8 Mar 22:58:28 ntpd[9031]: ntpd 4.2.8p12@1.3728-o (1): Starting 2022-03-08T22:58:28.528 INFO:teuthology.orchestra.run.smithi135.stdout: 8 Mar 22:58:28 ntpd[9031]: Command line: ntpd -gq 2022-03-08T22:58:28.529 INFO:teuthology.orchestra.run.smithi135.stdout: 8 Mar 22:58:28 ntpd[9031]: proto: precision = 0.058 usec (-24) 2022-03-08T22:58:28.529 INFO:teuthology.orchestra.run.smithi077.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-03-08T22:58:28.529 INFO:teuthology.orchestra.run.smithi135.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-03-08T22:58:28.530 INFO:teuthology.orchestra.run.smithi077.stdout: 8 Mar 22:58:28 ntpd[9066]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-03-08T22:58:28.530 INFO:teuthology.orchestra.run.smithi135.stdout: 8 Mar 22:58:28 ntpd[9031]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-03-08T22:58:28.531 INFO:teuthology.orchestra.run.smithi077.stderr:restrict ::: KOD does nothing without LIMITED. 2022-03-08T22:58:28.531 INFO:teuthology.orchestra.run.smithi135.stderr:restrict ::: KOD does nothing without LIMITED. 2022-03-08T22:58:28.533 INFO:teuthology.orchestra.run.smithi077.stdout: 8 Mar 22:58:28 ntpd[9066]: restrict ::: KOD does nothing without LIMITED. 2022-03-08T22:58:28.534 INFO:teuthology.orchestra.run.smithi077.stdout: 8 Mar 22:58:28 ntpd[9066]: Listen and drop on 0 v6wildcard [::]:123 2022-03-08T22:58:28.534 INFO:teuthology.orchestra.run.smithi077.stdout: 8 Mar 22:58:28 ntpd[9066]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2022-03-08T22:58:28.534 INFO:teuthology.orchestra.run.smithi077.stdout: 8 Mar 22:58:28 ntpd[9066]: Listen normally on 2 lo 127.0.0.1:123 2022-03-08T22:58:28.534 INFO:teuthology.orchestra.run.smithi077.stdout: 8 Mar 22:58:28 ntpd[9066]: Listen normally on 3 enp3s0f1 172.21.15.77:123 2022-03-08T22:58:28.535 INFO:teuthology.orchestra.run.smithi077.stdout: 8 Mar 22:58:28 ntpd[9066]: Listen normally on 4 lo [::1]:123 2022-03-08T22:58:28.535 INFO:teuthology.orchestra.run.smithi077.stdout: 8 Mar 22:58:28 ntpd[9066]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:d3d3%5]:123 2022-03-08T22:58:28.535 INFO:teuthology.orchestra.run.smithi077.stdout: 8 Mar 22:58:28 ntpd[9066]: Listening on routing socket on fd #22 for interface updates 2022-03-08T22:58:28.535 INFO:teuthology.orchestra.run.smithi135.stdout: 8 Mar 22:58:28 ntpd[9031]: restrict ::: KOD does nothing without LIMITED. 2022-03-08T22:58:28.536 INFO:teuthology.orchestra.run.smithi135.stdout: 8 Mar 22:58:28 ntpd[9031]: Listen and drop on 0 v6wildcard [::]:123 2022-03-08T22:58:28.536 INFO:teuthology.orchestra.run.smithi135.stdout: 8 Mar 22:58:28 ntpd[9031]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2022-03-08T22:58:28.536 INFO:teuthology.orchestra.run.smithi135.stdout: 8 Mar 22:58:28 ntpd[9031]: Listen normally on 2 lo 127.0.0.1:123 2022-03-08T22:58:28.536 INFO:teuthology.orchestra.run.smithi135.stdout: 8 Mar 22:58:28 ntpd[9031]: Listen normally on 3 enp3s0f1 172.21.15.135:123 2022-03-08T22:58:28.536 INFO:teuthology.orchestra.run.smithi135.stdout: 8 Mar 22:58:28 ntpd[9031]: Listen normally on 4 lo [::1]:123 2022-03-08T22:58:28.537 INFO:teuthology.orchestra.run.smithi135.stdout: 8 Mar 22:58:28 ntpd[9031]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:8083%5]:123 2022-03-08T22:58:28.537 INFO:teuthology.orchestra.run.smithi135.stdout: 8 Mar 22:58:28 ntpd[9031]: Listening on routing socket on fd #22 for interface updates 2022-03-08T22:58:29.522 INFO:teuthology.orchestra.run.smithi077.stderr: 8 Mar 22:58:29 ntpd[9066]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:58:29.522 INFO:teuthology.orchestra.run.smithi077.stderr: 8 Mar 22:58:29 ntpd[9066]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:58:29.522 INFO:teuthology.orchestra.run.smithi077.stderr: 8 Mar 22:58:29 ntpd[9066]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:58:29.523 INFO:teuthology.orchestra.run.smithi077.stderr: 8 Mar 22:58:29 ntpd[9066]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:58:29.523 INFO:teuthology.orchestra.run.smithi077.stderr: 8 Mar 22:58:29 ntpd[9066]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T22:58:29.523 INFO:teuthology.orchestra.run.smithi077.stderr: 8 Mar 22:58:29 ntpd[9066]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T22:58:29.527 INFO:teuthology.orchestra.run.smithi135.stderr: 8 Mar 22:58:29 ntpd[9031]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:58:29.527 INFO:teuthology.orchestra.run.smithi135.stderr: 8 Mar 22:58:29 ntpd[9031]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:58:29.528 INFO:teuthology.orchestra.run.smithi135.stderr: 8 Mar 22:58:29 ntpd[9031]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:58:29.528 INFO:teuthology.orchestra.run.smithi135.stderr: 8 Mar 22:58:29 ntpd[9031]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:58:29.528 INFO:teuthology.orchestra.run.smithi135.stderr: 8 Mar 22:58:29 ntpd[9031]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T22:58:29.528 INFO:teuthology.orchestra.run.smithi135.stderr: 8 Mar 22:58:29 ntpd[9031]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T22:58:30.521 INFO:teuthology.orchestra.run.smithi106.stderr: 8 Mar 22:58:30 ntpd[9027]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:58:30.521 INFO:teuthology.orchestra.run.smithi106.stderr: 8 Mar 22:58:30 ntpd[9027]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:58:30.522 INFO:teuthology.orchestra.run.smithi106.stderr: 8 Mar 22:58:30 ntpd[9027]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:58:30.522 INFO:teuthology.orchestra.run.smithi106.stderr: 8 Mar 22:58:30 ntpd[9027]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:58:30.522 INFO:teuthology.orchestra.run.smithi106.stderr: 8 Mar 22:58:30 ntpd[9027]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T22:58:30.522 INFO:teuthology.orchestra.run.smithi106.stderr: 8 Mar 22:58:30 ntpd[9027]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T22:58:30.523 INFO:teuthology.orchestra.run.smithi077.stderr: 8 Mar 22:58:30 ntpd[9066]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:58:30.524 INFO:teuthology.orchestra.run.smithi077.stderr: 8 Mar 22:58:30 ntpd[9066]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:58:30.524 INFO:teuthology.orchestra.run.smithi077.stderr: 8 Mar 22:58:30 ntpd[9066]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:58:30.524 INFO:teuthology.orchestra.run.smithi077.stderr: 8 Mar 22:58:30 ntpd[9066]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:58:30.524 INFO:teuthology.orchestra.run.smithi077.stderr: 8 Mar 22:58:30 ntpd[9066]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T22:58:30.525 INFO:teuthology.orchestra.run.smithi077.stderr: 8 Mar 22:58:30 ntpd[9066]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T22:58:30.527 INFO:teuthology.orchestra.run.smithi135.stderr: 8 Mar 22:58:30 ntpd[9031]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:58:30.527 INFO:teuthology.orchestra.run.smithi135.stderr: 8 Mar 22:58:30 ntpd[9031]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:58:30.528 INFO:teuthology.orchestra.run.smithi135.stderr: 8 Mar 22:58:30 ntpd[9031]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:58:30.528 INFO:teuthology.orchestra.run.smithi135.stderr: 8 Mar 22:58:30 ntpd[9031]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:58:30.528 INFO:teuthology.orchestra.run.smithi135.stderr: 8 Mar 22:58:30 ntpd[9031]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T22:58:30.528 INFO:teuthology.orchestra.run.smithi135.stderr: 8 Mar 22:58:30 ntpd[9031]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T22:58:31.521 INFO:teuthology.orchestra.run.smithi106.stderr: 8 Mar 22:58:31 ntpd[9027]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:58:31.522 INFO:teuthology.orchestra.run.smithi106.stderr: 8 Mar 22:58:31 ntpd[9027]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:58:31.522 INFO:teuthology.orchestra.run.smithi106.stderr: 8 Mar 22:58:31 ntpd[9027]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:58:31.522 INFO:teuthology.orchestra.run.smithi106.stderr: 8 Mar 22:58:31 ntpd[9027]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:58:31.522 INFO:teuthology.orchestra.run.smithi106.stderr: 8 Mar 22:58:31 ntpd[9027]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T22:58:31.522 INFO:teuthology.orchestra.run.smithi106.stderr: 8 Mar 22:58:31 ntpd[9027]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T22:58:31.524 INFO:teuthology.orchestra.run.smithi077.stderr: 8 Mar 22:58:31 ntpd[9066]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:58:31.524 INFO:teuthology.orchestra.run.smithi077.stderr: 8 Mar 22:58:31 ntpd[9066]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:58:31.525 INFO:teuthology.orchestra.run.smithi077.stderr: 8 Mar 22:58:31 ntpd[9066]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:58:31.525 INFO:teuthology.orchestra.run.smithi077.stderr: 8 Mar 22:58:31 ntpd[9066]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:58:31.525 INFO:teuthology.orchestra.run.smithi077.stderr: 8 Mar 22:58:31 ntpd[9066]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:58:31.525 INFO:teuthology.orchestra.run.smithi077.stderr: 8 Mar 22:58:31 ntpd[9066]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:58:31.526 INFO:teuthology.orchestra.run.smithi077.stderr: 8 Mar 22:58:31 ntpd[9066]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T22:58:31.526 INFO:teuthology.orchestra.run.smithi077.stderr: 8 Mar 22:58:31 ntpd[9066]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T22:58:31.526 INFO:teuthology.orchestra.run.smithi077.stderr: 8 Mar 22:58:31 ntpd[9066]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:58:31.526 INFO:teuthology.orchestra.run.smithi077.stderr: 8 Mar 22:58:31 ntpd[9066]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:58:31.527 INFO:teuthology.orchestra.run.smithi077.stderr: 8 Mar 22:58:31 ntpd[9066]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T22:58:31.527 INFO:teuthology.orchestra.run.smithi077.stderr: 8 Mar 22:58:31 ntpd[9066]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T22:58:31.528 INFO:teuthology.orchestra.run.smithi135.stderr: 8 Mar 22:58:31 ntpd[9031]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:58:31.528 INFO:teuthology.orchestra.run.smithi135.stderr: 8 Mar 22:58:31 ntpd[9031]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:58:31.529 INFO:teuthology.orchestra.run.smithi135.stderr: 8 Mar 22:58:31 ntpd[9031]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:58:31.529 INFO:teuthology.orchestra.run.smithi135.stderr: 8 Mar 22:58:31 ntpd[9031]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:58:31.529 INFO:teuthology.orchestra.run.smithi135.stderr: 8 Mar 22:58:31 ntpd[9031]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T22:58:31.529 INFO:teuthology.orchestra.run.smithi135.stderr: 8 Mar 22:58:31 ntpd[9031]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T22:58:32.525 INFO:teuthology.orchestra.run.smithi077.stderr: 8 Mar 22:58:32 ntpd[9066]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:58:32.525 INFO:teuthology.orchestra.run.smithi077.stderr: 8 Mar 22:58:32 ntpd[9066]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:58:32.525 INFO:teuthology.orchestra.run.smithi077.stderr: 8 Mar 22:58:32 ntpd[9066]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:58:32.526 INFO:teuthology.orchestra.run.smithi077.stderr: 8 Mar 22:58:32 ntpd[9066]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:58:32.526 INFO:teuthology.orchestra.run.smithi077.stderr: 8 Mar 22:58:32 ntpd[9066]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:58:32.526 INFO:teuthology.orchestra.run.smithi077.stderr: 8 Mar 22:58:32 ntpd[9066]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:58:32.526 INFO:teuthology.orchestra.run.smithi077.stderr: 8 Mar 22:58:32 ntpd[9066]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T22:58:32.527 INFO:teuthology.orchestra.run.smithi077.stderr: 8 Mar 22:58:32 ntpd[9066]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T22:58:32.527 INFO:teuthology.orchestra.run.smithi077.stderr: 8 Mar 22:58:32 ntpd[9066]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:58:32.527 INFO:teuthology.orchestra.run.smithi077.stderr: 8 Mar 22:58:32 ntpd[9066]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:58:32.527 INFO:teuthology.orchestra.run.smithi077.stderr: 8 Mar 22:58:32 ntpd[9066]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T22:58:32.528 INFO:teuthology.orchestra.run.smithi077.stderr: 8 Mar 22:58:32 ntpd[9066]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T22:58:32.528 INFO:teuthology.orchestra.run.smithi106.stderr: 8 Mar 22:58:32 ntpd[9027]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:58:32.528 INFO:teuthology.orchestra.run.smithi106.stderr: 8 Mar 22:58:32 ntpd[9027]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:58:32.528 INFO:teuthology.orchestra.run.smithi106.stderr: 8 Mar 22:58:32 ntpd[9027]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:58:32.529 INFO:teuthology.orchestra.run.smithi106.stderr: 8 Mar 22:58:32 ntpd[9027]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:58:32.529 INFO:teuthology.orchestra.run.smithi106.stderr: 8 Mar 22:58:32 ntpd[9027]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:58:32.529 INFO:teuthology.orchestra.run.smithi106.stderr: 8 Mar 22:58:32 ntpd[9027]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:58:32.529 INFO:teuthology.orchestra.run.smithi106.stderr: 8 Mar 22:58:32 ntpd[9027]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T22:58:32.530 INFO:teuthology.orchestra.run.smithi106.stderr: 8 Mar 22:58:32 ntpd[9027]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T22:58:32.530 INFO:teuthology.orchestra.run.smithi106.stderr: 8 Mar 22:58:32 ntpd[9027]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:58:32.530 INFO:teuthology.orchestra.run.smithi106.stderr: 8 Mar 22:58:32 ntpd[9027]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:58:32.530 INFO:teuthology.orchestra.run.smithi106.stderr: 8 Mar 22:58:32 ntpd[9027]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T22:58:32.531 INFO:teuthology.orchestra.run.smithi106.stderr: 8 Mar 22:58:32 ntpd[9027]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T22:58:32.534 INFO:teuthology.orchestra.run.smithi135.stderr: 8 Mar 22:58:32 ntpd[9031]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:58:32.534 INFO:teuthology.orchestra.run.smithi135.stderr: 8 Mar 22:58:32 ntpd[9031]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:58:32.534 INFO:teuthology.orchestra.run.smithi135.stderr: 8 Mar 22:58:32 ntpd[9031]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:58:32.535 INFO:teuthology.orchestra.run.smithi135.stderr: 8 Mar 22:58:32 ntpd[9031]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:58:32.535 INFO:teuthology.orchestra.run.smithi135.stderr: 8 Mar 22:58:32 ntpd[9031]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:58:32.535 INFO:teuthology.orchestra.run.smithi135.stderr: 8 Mar 22:58:32 ntpd[9031]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:58:32.535 INFO:teuthology.orchestra.run.smithi135.stderr: 8 Mar 22:58:32 ntpd[9031]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:58:32.535 INFO:teuthology.orchestra.run.smithi135.stderr: 8 Mar 22:58:32 ntpd[9031]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:58:32.536 INFO:teuthology.orchestra.run.smithi135.stderr: 8 Mar 22:58:32 ntpd[9031]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T22:58:32.536 INFO:teuthology.orchestra.run.smithi135.stderr: 8 Mar 22:58:32 ntpd[9031]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T22:58:32.536 INFO:teuthology.orchestra.run.smithi135.stderr: 8 Mar 22:58:32 ntpd[9031]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:58:32.536 INFO:teuthology.orchestra.run.smithi135.stderr: 8 Mar 22:58:32 ntpd[9031]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:58:32.537 INFO:teuthology.orchestra.run.smithi135.stderr: 8 Mar 22:58:32 ntpd[9031]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T22:58:32.537 INFO:teuthology.orchestra.run.smithi135.stderr: 8 Mar 22:58:32 ntpd[9031]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T22:58:32.537 INFO:teuthology.orchestra.run.smithi135.stderr: 8 Mar 22:58:32 ntpd[9031]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:58:32.537 INFO:teuthology.orchestra.run.smithi135.stderr: 8 Mar 22:58:32 ntpd[9031]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:58:32.538 INFO:teuthology.orchestra.run.smithi135.stderr: 8 Mar 22:58:32 ntpd[9031]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T22:58:32.538 INFO:teuthology.orchestra.run.smithi135.stderr: 8 Mar 22:58:32 ntpd[9031]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T22:58:33.521 INFO:teuthology.orchestra.run.smithi106.stderr: 8 Mar 22:58:33 ntpd[9027]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:58:33.522 INFO:teuthology.orchestra.run.smithi106.stderr: 8 Mar 22:58:33 ntpd[9027]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:58:33.522 INFO:teuthology.orchestra.run.smithi106.stderr: 8 Mar 22:58:33 ntpd[9027]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:58:33.523 INFO:teuthology.orchestra.run.smithi106.stderr: 8 Mar 22:58:33 ntpd[9027]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:58:33.523 INFO:teuthology.orchestra.run.smithi106.stderr: 8 Mar 22:58:33 ntpd[9027]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:58:33.523 INFO:teuthology.orchestra.run.smithi106.stderr: 8 Mar 22:58:33 ntpd[9027]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:58:33.523 INFO:teuthology.orchestra.run.smithi106.stderr: 8 Mar 22:58:33 ntpd[9027]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T22:58:33.524 INFO:teuthology.orchestra.run.smithi106.stderr: 8 Mar 22:58:33 ntpd[9027]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T22:58:33.524 INFO:teuthology.orchestra.run.smithi106.stderr: 8 Mar 22:58:33 ntpd[9027]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:58:33.524 INFO:teuthology.orchestra.run.smithi106.stderr: 8 Mar 22:58:33 ntpd[9027]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:58:33.524 INFO:teuthology.orchestra.run.smithi106.stderr: 8 Mar 22:58:33 ntpd[9027]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T22:58:33.525 INFO:teuthology.orchestra.run.smithi106.stderr: 8 Mar 22:58:33 ntpd[9027]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T22:58:33.527 INFO:teuthology.orchestra.run.smithi077.stderr: 8 Mar 22:58:33 ntpd[9066]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:58:33.527 INFO:teuthology.orchestra.run.smithi077.stderr: 8 Mar 22:58:33 ntpd[9066]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:58:33.527 INFO:teuthology.orchestra.run.smithi077.stderr: 8 Mar 22:58:33 ntpd[9066]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:58:33.528 INFO:teuthology.orchestra.run.smithi077.stderr: 8 Mar 22:58:33 ntpd[9066]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:58:33.528 INFO:teuthology.orchestra.run.smithi077.stderr: 8 Mar 22:58:33 ntpd[9066]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:58:33.528 INFO:teuthology.orchestra.run.smithi077.stderr: 8 Mar 22:58:33 ntpd[9066]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:58:33.528 INFO:teuthology.orchestra.run.smithi077.stderr: 8 Mar 22:58:33 ntpd[9066]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T22:58:33.528 INFO:teuthology.orchestra.run.smithi077.stderr: 8 Mar 22:58:33 ntpd[9066]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T22:58:33.529 INFO:teuthology.orchestra.run.smithi077.stderr: 8 Mar 22:58:33 ntpd[9066]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:58:33.529 INFO:teuthology.orchestra.run.smithi077.stderr: 8 Mar 22:58:33 ntpd[9066]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:58:33.529 INFO:teuthology.orchestra.run.smithi077.stderr: 8 Mar 22:58:33 ntpd[9066]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T22:58:33.529 INFO:teuthology.orchestra.run.smithi077.stderr: 8 Mar 22:58:33 ntpd[9066]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T22:58:33.530 INFO:teuthology.orchestra.run.smithi135.stderr: 8 Mar 22:58:33 ntpd[9031]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:58:33.531 INFO:teuthology.orchestra.run.smithi135.stderr: 8 Mar 22:58:33 ntpd[9031]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:58:33.531 INFO:teuthology.orchestra.run.smithi135.stderr: 8 Mar 22:58:33 ntpd[9031]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:58:33.531 INFO:teuthology.orchestra.run.smithi135.stderr: 8 Mar 22:58:33 ntpd[9031]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:58:33.531 INFO:teuthology.orchestra.run.smithi135.stderr: 8 Mar 22:58:33 ntpd[9031]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T22:58:33.532 INFO:teuthology.orchestra.run.smithi135.stderr: 8 Mar 22:58:33 ntpd[9031]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T22:58:34.522 INFO:teuthology.orchestra.run.smithi106.stderr: 8 Mar 22:58:34 ntpd[9027]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:58:34.522 INFO:teuthology.orchestra.run.smithi106.stderr: 8 Mar 22:58:34 ntpd[9027]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:58:34.522 INFO:teuthology.orchestra.run.smithi106.stderr: 8 Mar 22:58:34 ntpd[9027]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:58:34.523 INFO:teuthology.orchestra.run.smithi106.stderr: 8 Mar 22:58:34 ntpd[9027]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:58:34.523 INFO:teuthology.orchestra.run.smithi106.stderr: 8 Mar 22:58:34 ntpd[9027]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:58:34.523 INFO:teuthology.orchestra.run.smithi106.stderr: 8 Mar 22:58:34 ntpd[9027]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:58:34.523 INFO:teuthology.orchestra.run.smithi106.stderr: 8 Mar 22:58:34 ntpd[9027]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T22:58:34.523 INFO:teuthology.orchestra.run.smithi106.stderr: 8 Mar 22:58:34 ntpd[9027]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T22:58:34.524 INFO:teuthology.orchestra.run.smithi106.stderr: 8 Mar 22:58:34 ntpd[9027]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:58:34.524 INFO:teuthology.orchestra.run.smithi106.stderr: 8 Mar 22:58:34 ntpd[9027]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:58:34.525 INFO:teuthology.orchestra.run.smithi106.stderr: 8 Mar 22:58:34 ntpd[9027]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T22:58:34.525 INFO:teuthology.orchestra.run.smithi106.stderr: 8 Mar 22:58:34 ntpd[9027]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T22:58:34.527 INFO:teuthology.orchestra.run.smithi077.stderr: 8 Mar 22:58:34 ntpd[9066]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:58:34.527 INFO:teuthology.orchestra.run.smithi077.stderr: 8 Mar 22:58:34 ntpd[9066]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:58:34.528 INFO:teuthology.orchestra.run.smithi077.stderr: 8 Mar 22:58:34 ntpd[9066]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:58:34.528 INFO:teuthology.orchestra.run.smithi077.stderr: 8 Mar 22:58:34 ntpd[9066]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:58:34.528 INFO:teuthology.orchestra.run.smithi077.stderr: 8 Mar 22:58:34 ntpd[9066]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:58:34.528 INFO:teuthology.orchestra.run.smithi077.stderr: 8 Mar 22:58:34 ntpd[9066]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:58:34.528 INFO:teuthology.orchestra.run.smithi077.stderr: 8 Mar 22:58:34 ntpd[9066]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T22:58:34.529 INFO:teuthology.orchestra.run.smithi077.stderr: 8 Mar 22:58:34 ntpd[9066]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T22:58:34.529 INFO:teuthology.orchestra.run.smithi077.stderr: 8 Mar 22:58:34 ntpd[9066]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:58:34.529 INFO:teuthology.orchestra.run.smithi077.stderr: 8 Mar 22:58:34 ntpd[9066]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:58:34.529 INFO:teuthology.orchestra.run.smithi077.stderr: 8 Mar 22:58:34 ntpd[9066]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T22:58:34.530 INFO:teuthology.orchestra.run.smithi077.stderr: 8 Mar 22:58:34 ntpd[9066]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T22:58:34.532 INFO:teuthology.orchestra.run.smithi135.stderr: 8 Mar 22:58:34 ntpd[9031]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:58:34.532 INFO:teuthology.orchestra.run.smithi135.stderr: 8 Mar 22:58:34 ntpd[9031]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:58:34.532 INFO:teuthology.orchestra.run.smithi135.stderr: 8 Mar 22:58:34 ntpd[9031]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:58:34.533 INFO:teuthology.orchestra.run.smithi135.stderr: 8 Mar 22:58:34 ntpd[9031]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:58:34.533 INFO:teuthology.orchestra.run.smithi135.stderr: 8 Mar 22:58:34 ntpd[9031]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:58:34.533 INFO:teuthology.orchestra.run.smithi135.stderr: 8 Mar 22:58:34 ntpd[9031]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:58:34.533 INFO:teuthology.orchestra.run.smithi135.stderr: 8 Mar 22:58:34 ntpd[9031]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:58:34.534 INFO:teuthology.orchestra.run.smithi135.stderr: 8 Mar 22:58:34 ntpd[9031]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:58:34.534 INFO:teuthology.orchestra.run.smithi135.stderr: 8 Mar 22:58:34 ntpd[9031]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T22:58:34.534 INFO:teuthology.orchestra.run.smithi135.stderr: 8 Mar 22:58:34 ntpd[9031]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T22:58:34.534 INFO:teuthology.orchestra.run.smithi135.stderr: 8 Mar 22:58:34 ntpd[9031]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:58:34.535 INFO:teuthology.orchestra.run.smithi135.stderr: 8 Mar 22:58:34 ntpd[9031]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:58:34.535 INFO:teuthology.orchestra.run.smithi135.stderr: 8 Mar 22:58:34 ntpd[9031]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T22:58:34.535 INFO:teuthology.orchestra.run.smithi135.stderr: 8 Mar 22:58:34 ntpd[9031]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T22:58:34.535 INFO:teuthology.orchestra.run.smithi135.stderr: 8 Mar 22:58:34 ntpd[9031]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:58:34.536 INFO:teuthology.orchestra.run.smithi135.stderr: 8 Mar 22:58:34 ntpd[9031]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:58:34.536 INFO:teuthology.orchestra.run.smithi135.stderr: 8 Mar 22:58:34 ntpd[9031]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T22:58:34.536 INFO:teuthology.orchestra.run.smithi135.stderr: 8 Mar 22:58:34 ntpd[9031]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T22:58:35.521 INFO:teuthology.orchestra.run.smithi106.stderr: 8 Mar 22:58:35 ntpd[9027]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:58:35.522 INFO:teuthology.orchestra.run.smithi106.stderr: 8 Mar 22:58:35 ntpd[9027]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:58:35.522 INFO:teuthology.orchestra.run.smithi106.stderr: 8 Mar 22:58:35 ntpd[9027]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:58:35.522 INFO:teuthology.orchestra.run.smithi106.stderr: 8 Mar 22:58:35 ntpd[9027]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:58:35.522 INFO:teuthology.orchestra.run.smithi106.stderr: 8 Mar 22:58:35 ntpd[9027]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:58:35.523 INFO:teuthology.orchestra.run.smithi106.stderr: 8 Mar 22:58:35 ntpd[9027]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:58:35.523 INFO:teuthology.orchestra.run.smithi106.stderr: 8 Mar 22:58:35 ntpd[9027]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T22:58:35.523 INFO:teuthology.orchestra.run.smithi106.stderr: 8 Mar 22:58:35 ntpd[9027]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T22:58:35.523 INFO:teuthology.orchestra.run.smithi106.stderr: 8 Mar 22:58:35 ntpd[9027]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:58:35.524 INFO:teuthology.orchestra.run.smithi106.stderr: 8 Mar 22:58:35 ntpd[9027]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:58:35.524 INFO:teuthology.orchestra.run.smithi106.stderr: 8 Mar 22:58:35 ntpd[9027]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T22:58:35.524 INFO:teuthology.orchestra.run.smithi106.stderr: 8 Mar 22:58:35 ntpd[9027]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T22:58:35.525 INFO:teuthology.orchestra.run.smithi077.stderr: 8 Mar 22:58:35 ntpd[9066]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:58:35.526 INFO:teuthology.orchestra.run.smithi077.stderr: 8 Mar 22:58:35 ntpd[9066]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:58:35.526 INFO:teuthology.orchestra.run.smithi077.stderr: 8 Mar 22:58:35 ntpd[9066]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:58:35.526 INFO:teuthology.orchestra.run.smithi077.stderr: 8 Mar 22:58:35 ntpd[9066]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:58:35.526 INFO:teuthology.orchestra.run.smithi077.stderr: 8 Mar 22:58:35 ntpd[9066]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:58:35.526 INFO:teuthology.orchestra.run.smithi077.stderr: 8 Mar 22:58:35 ntpd[9066]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:58:35.527 INFO:teuthology.orchestra.run.smithi077.stderr: 8 Mar 22:58:35 ntpd[9066]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T22:58:35.527 INFO:teuthology.orchestra.run.smithi077.stderr: 8 Mar 22:58:35 ntpd[9066]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T22:58:35.527 INFO:teuthology.orchestra.run.smithi077.stdout: 8 Mar 22:58:35 ntpd[9066]: ntpd: time slew +0.000383 s 2022-03-08T22:58:35.528 INFO:teuthology.orchestra.run.smithi077.stdout:ntpd: time slew +0.000383s 2022-03-08T22:58:35.528 INFO:teuthology.orchestra.run.smithi077.stderr: 8 Mar 22:58:35 ntpd[9066]: can't open /var/log/ntpstats/loopstats.20220308: Permission denied 2022-03-08T22:58:35.529 INFO:teuthology.orchestra.run.smithi135.stderr: 8 Mar 22:58:35 ntpd[9031]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:58:35.529 INFO:teuthology.orchestra.run.smithi135.stderr: 8 Mar 22:58:35 ntpd[9031]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:58:35.530 INFO:teuthology.orchestra.run.smithi135.stderr: 8 Mar 22:58:35 ntpd[9031]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:58:35.530 INFO:teuthology.orchestra.run.smithi135.stderr: 8 Mar 22:58:35 ntpd[9031]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:58:35.530 INFO:teuthology.orchestra.run.smithi135.stderr: 8 Mar 22:58:35 ntpd[9031]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T22:58:35.530 INFO:teuthology.orchestra.run.smithi135.stderr: 8 Mar 22:58:35 ntpd[9031]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T22:58:35.531 INFO:teuthology.orchestra.run.smithi135.stdout: 8 Mar 22:58:35 ntpd[9031]: ntpd: time slew -0.001678 s 2022-03-08T22:58:35.531 INFO:teuthology.orchestra.run.smithi135.stdout:ntpd: time slew -0.001678s 2022-03-08T22:58:35.531 INFO:teuthology.orchestra.run.smithi135.stderr: 8 Mar 22:58:35 ntpd[9031]: can't open /var/log/ntpstats/loopstats.20220308: Permission denied 2022-03-08T22:58:35.578 INFO:teuthology.orchestra.run.smithi135.stdout: remote refid st t when poll reach delay offset jitter 2022-03-08T22:58:35.579 INFO:teuthology.orchestra.run.smithi135.stdout:============================================================================== 2022-03-08T22:58:35.579 INFO:teuthology.orchestra.run.smithi135.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-08T22:58:35.579 INFO:teuthology.orchestra.run.smithi135.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-08T22:58:35.579 INFO:teuthology.orchestra.run.smithi135.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-08T22:58:35.580 INFO:teuthology.orchestra.run.smithi135.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-08T22:58:35.587 INFO:teuthology.orchestra.run.smithi077.stdout: remote refid st t when poll reach delay offset jitter 2022-03-08T22:58:35.587 INFO:teuthology.orchestra.run.smithi077.stdout:============================================================================== 2022-03-08T22:58:35.587 INFO:teuthology.orchestra.run.smithi077.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-08T22:58:35.587 INFO:teuthology.orchestra.run.smithi077.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-08T22:58:35.588 INFO:teuthology.orchestra.run.smithi077.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-08T22:58:35.588 INFO:teuthology.orchestra.run.smithi077.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-08T22:58:36.521 INFO:teuthology.orchestra.run.smithi106.stderr: 8 Mar 22:58:36 ntpd[9027]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:58:36.522 INFO:teuthology.orchestra.run.smithi106.stderr: 8 Mar 22:58:36 ntpd[9027]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:58:36.522 INFO:teuthology.orchestra.run.smithi106.stderr: 8 Mar 22:58:36 ntpd[9027]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:58:36.522 INFO:teuthology.orchestra.run.smithi106.stderr: 8 Mar 22:58:36 ntpd[9027]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:58:36.523 INFO:teuthology.orchestra.run.smithi106.stderr: 8 Mar 22:58:36 ntpd[9027]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:58:36.523 INFO:teuthology.orchestra.run.smithi106.stderr: 8 Mar 22:58:36 ntpd[9027]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:58:36.523 INFO:teuthology.orchestra.run.smithi106.stderr: 8 Mar 22:58:36 ntpd[9027]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T22:58:36.523 INFO:teuthology.orchestra.run.smithi106.stderr: 8 Mar 22:58:36 ntpd[9027]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T22:58:36.523 INFO:teuthology.orchestra.run.smithi106.stderr: 8 Mar 22:58:36 ntpd[9027]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:58:36.524 INFO:teuthology.orchestra.run.smithi106.stderr: 8 Mar 22:58:36 ntpd[9027]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:58:36.524 INFO:teuthology.orchestra.run.smithi106.stderr: 8 Mar 22:58:36 ntpd[9027]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T22:58:36.524 INFO:teuthology.orchestra.run.smithi106.stderr: 8 Mar 22:58:36 ntpd[9027]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T22:58:36.525 INFO:teuthology.orchestra.run.smithi106.stdout: 8 Mar 22:58:36 ntpd[9027]: ntpd: time slew +0.008573 s 2022-03-08T22:58:36.525 INFO:teuthology.orchestra.run.smithi106.stdout:ntpd: time slew +0.008573s 2022-03-08T22:58:36.525 INFO:teuthology.orchestra.run.smithi106.stderr: 8 Mar 22:58:36 ntpd[9027]: can't open /var/log/ntpstats/loopstats.20220308: Permission denied 2022-03-08T22:58:36.581 INFO:teuthology.orchestra.run.smithi106.stdout: remote refid st t when poll reach delay offset jitter 2022-03-08T22:58:36.581 INFO:teuthology.orchestra.run.smithi106.stdout:============================================================================== 2022-03-08T22:58:36.581 INFO:teuthology.orchestra.run.smithi106.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-08T22:58:36.582 INFO:teuthology.orchestra.run.smithi106.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-08T22:58:36.582 INFO:teuthology.orchestra.run.smithi106.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-08T22:58:36.582 INFO:teuthology.orchestra.run.smithi106.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-08T22:58:36.583 INFO:teuthology.run_tasks:Running task kubeadm... 2022-03-08T22:58:36.599 INFO:tasks.kubeadm:Kubeadm start 2022-03-08T22:58:36.600 INFO:tasks.kubeadm:Config: {'pod_network': 'flannel', 'version': '1.21'} 2022-03-08T22:58:36.600 DEBUG:teuthology.orchestra.run.smithi077:> sudo modprobe br_netfilter && sudo sysctl net.bridge.bridge-nf-call-ip6tables=1 && sudo sysctl net.bridge.bridge-nf-call-iptables=1 && sudo sysctl net.ipv4.ip_forward=1 && sudo swapoff -a 2022-03-08T22:58:36.604 DEBUG:teuthology.orchestra.run.smithi106:> sudo modprobe br_netfilter && sudo sysctl net.bridge.bridge-nf-call-ip6tables=1 && sudo sysctl net.bridge.bridge-nf-call-iptables=1 && sudo sysctl net.ipv4.ip_forward=1 && sudo swapoff -a 2022-03-08T22:58:36.606 DEBUG:teuthology.orchestra.run.smithi135:> sudo modprobe br_netfilter && sudo sysctl net.bridge.bridge-nf-call-ip6tables=1 && sudo sysctl net.bridge.bridge-nf-call-iptables=1 && sudo sysctl net.ipv4.ip_forward=1 && sudo swapoff -a 2022-03-08T22:58:36.628 INFO:teuthology.orchestra.run.smithi077.stdout:net.bridge.bridge-nf-call-ip6tables = 1 2022-03-08T22:58:36.629 INFO:teuthology.orchestra.run.smithi106.stdout:net.bridge.bridge-nf-call-ip6tables = 1 2022-03-08T22:58:36.630 INFO:teuthology.orchestra.run.smithi135.stdout:net.bridge.bridge-nf-call-ip6tables = 1 2022-03-08T22:58:36.635 INFO:teuthology.orchestra.run.smithi077.stdout:net.bridge.bridge-nf-call-iptables = 1 2022-03-08T22:58:36.638 INFO:teuthology.orchestra.run.smithi106.stdout:net.bridge.bridge-nf-call-iptables = 1 2022-03-08T22:58:36.639 INFO:teuthology.orchestra.run.smithi135.stdout:net.bridge.bridge-nf-call-iptables = 1 2022-03-08T22:58:36.644 INFO:teuthology.orchestra.run.smithi077.stdout:net.ipv4.ip_forward = 1 2022-03-08T22:58:36.645 INFO:teuthology.orchestra.run.smithi106.stdout:net.ipv4.ip_forward = 1 2022-03-08T22:58:36.648 INFO:teuthology.orchestra.run.smithi135.stdout:net.ipv4.ip_forward = 1 2022-03-08T22:58:36.712 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2022-03-08T22:58:36.713 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/etc/docker/daemon.json of=/dev/stdout 2022-03-08T22:58:36.729 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2022-03-08T22:58:36.729 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/etc/docker/daemon.json 2022-03-08T22:58:36.786 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2022-03-08T22:58:36.786 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd if=/etc/docker/daemon.json of=/dev/stdout 2022-03-08T22:58:36.802 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2022-03-08T22:58:36.802 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd of=/etc/docker/daemon.json 2022-03-08T22:58:36.860 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2022-03-08T22:58:36.860 DEBUG:teuthology.orchestra.run.smithi135:> sudo dd if=/etc/docker/daemon.json of=/dev/stdout 2022-03-08T22:58:36.877 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2022-03-08T22:58:36.878 DEBUG:teuthology.orchestra.run.smithi135:> sudo dd of=/etc/docker/daemon.json 2022-03-08T22:58:36.932 DEBUG:teuthology.orchestra.run.smithi077:> sudo systemctl restart docker || true 2022-03-08T22:58:36.935 DEBUG:teuthology.orchestra.run.smithi106:> sudo systemctl restart docker || true 2022-03-08T22:58:36.938 DEBUG:teuthology.orchestra.run.smithi135:> sudo systemctl restart docker || true 2022-03-08T22:58:37.909 INFO:tasks.kubeadm:Identifying pod subnet 2022-03-08T22:58:37.917 INFO:tasks.kubeadm:172.21.15.77 is in 172.21.0.0/20 at pos 3916 2022-03-08T22:58:37.930 INFO:tasks.kubeadm:Pod subnet: 10.250.96.0/21 2022-03-08T22:58:37.931 INFO:tasks.kubeadm:Installing kube{adm,ctl,let} 2022-03-08T22:58:37.931 DEBUG:teuthology.orchestra.run.smithi077:> sudo apt update && sudo apt install -y apt-transport-https ca-certificates curl && sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg && echo 'deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main' | sudo tee /etc/apt/sources.list.d/kubernetes.list && sudo apt update && sudo apt install -y kubelet kubeadm kubectl bridge-utils 2022-03-08T22:58:37.934 DEBUG:teuthology.orchestra.run.smithi106:> sudo apt update && sudo apt install -y apt-transport-https ca-certificates curl && sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg && echo 'deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main' | sudo tee /etc/apt/sources.list.d/kubernetes.list && sudo apt update && sudo apt install -y kubelet kubeadm kubectl bridge-utils 2022-03-08T22:58:37.937 DEBUG:teuthology.orchestra.run.smithi135:> sudo apt update && sudo apt install -y apt-transport-https ca-certificates curl && sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg && echo 'deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main' | sudo tee /etc/apt/sources.list.d/kubernetes.list && sudo apt update && sudo apt install -y kubelet kubeadm kubectl bridge-utils 2022-03-08T22:58:37.962 INFO:teuthology.orchestra.run.smithi106.stderr: 2022-03-08T22:58:37.962 INFO:teuthology.orchestra.run.smithi106.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-08T22:58:37.962 INFO:teuthology.orchestra.run.smithi106.stderr: 2022-03-08T22:58:37.963 INFO:teuthology.orchestra.run.smithi135.stderr: 2022-03-08T22:58:37.963 INFO:teuthology.orchestra.run.smithi135.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-08T22:58:37.964 INFO:teuthology.orchestra.run.smithi135.stderr: 2022-03-08T22:58:37.964 INFO:teuthology.orchestra.run.smithi077.stderr: 2022-03-08T22:58:37.965 INFO:teuthology.orchestra.run.smithi077.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-08T22:58:37.965 INFO:teuthology.orchestra.run.smithi077.stderr: 2022-03-08T22:58:38.080 INFO:teuthology.orchestra.run.smithi135.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-03-08T22:58:38.081 INFO:teuthology.orchestra.run.smithi077.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-03-08T22:58:38.193 INFO:teuthology.orchestra.run.smithi135.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-08T22:58:38.193 INFO:teuthology.orchestra.run.smithi077.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-08T22:58:38.201 INFO:teuthology.orchestra.run.smithi106.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-08T22:58:38.202 INFO:teuthology.orchestra.run.smithi106.stdout:Hit:2 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-03-08T22:58:38.273 INFO:teuthology.orchestra.run.smithi106.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-03-08T22:58:38.275 INFO:teuthology.orchestra.run.smithi135.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-03-08T22:58:38.275 INFO:teuthology.orchestra.run.smithi077.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-03-08T22:58:38.354 INFO:teuthology.orchestra.run.smithi106.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-03-08T22:58:38.357 INFO:teuthology.orchestra.run.smithi135.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-03-08T22:58:38.357 INFO:teuthology.orchestra.run.smithi077.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-03-08T22:58:39.553 INFO:teuthology.orchestra.run.smithi135.stdout:Reading package lists... 2022-03-08T22:58:39.559 INFO:teuthology.orchestra.run.smithi077.stdout:Reading package lists... 2022-03-08T22:58:39.660 INFO:teuthology.orchestra.run.smithi106.stdout:Reading package lists... 2022-03-08T22:58:39.724 INFO:teuthology.orchestra.run.smithi135.stdout:Building dependency tree... 2022-03-08T22:58:39.725 INFO:teuthology.orchestra.run.smithi135.stdout:Reading state information... 2022-03-08T22:58:39.729 INFO:teuthology.orchestra.run.smithi077.stdout:Building dependency tree... 2022-03-08T22:58:39.730 INFO:teuthology.orchestra.run.smithi077.stdout:Reading state information... 2022-03-08T22:58:39.815 INFO:teuthology.orchestra.run.smithi135.stdout:85 packages can be upgraded. Run 'apt list --upgradable' to see them. 2022-03-08T22:58:39.825 INFO:teuthology.orchestra.run.smithi077.stdout:85 packages can be upgraded. Run 'apt list --upgradable' to see them. 2022-03-08T22:58:39.829 INFO:teuthology.orchestra.run.smithi135.stderr: 2022-03-08T22:58:39.830 INFO:teuthology.orchestra.run.smithi135.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-08T22:58:39.830 INFO:teuthology.orchestra.run.smithi135.stderr: 2022-03-08T22:58:39.835 INFO:teuthology.orchestra.run.smithi106.stdout:Building dependency tree... 2022-03-08T22:58:39.836 INFO:teuthology.orchestra.run.smithi106.stdout:Reading state information... 2022-03-08T22:58:39.837 INFO:teuthology.orchestra.run.smithi077.stderr: 2022-03-08T22:58:39.837 INFO:teuthology.orchestra.run.smithi077.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-08T22:58:39.838 INFO:teuthology.orchestra.run.smithi077.stderr: 2022-03-08T22:58:39.883 INFO:teuthology.orchestra.run.smithi135.stdout:Reading package lists... 2022-03-08T22:58:39.890 INFO:teuthology.orchestra.run.smithi077.stdout:Reading package lists... 2022-03-08T22:58:39.930 INFO:teuthology.orchestra.run.smithi106.stdout:85 packages can be upgraded. Run 'apt list --upgradable' to see them. 2022-03-08T22:58:39.946 INFO:teuthology.orchestra.run.smithi106.stderr: 2022-03-08T22:58:39.947 INFO:teuthology.orchestra.run.smithi106.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-08T22:58:39.947 INFO:teuthology.orchestra.run.smithi106.stderr: 2022-03-08T22:58:39.999 INFO:teuthology.orchestra.run.smithi106.stdout:Reading package lists... 2022-03-08T22:58:40.059 INFO:teuthology.orchestra.run.smithi135.stdout:Building dependency tree... 2022-03-08T22:58:40.060 INFO:teuthology.orchestra.run.smithi135.stdout:Reading state information... 2022-03-08T22:58:40.066 INFO:teuthology.orchestra.run.smithi077.stdout:Building dependency tree... 2022-03-08T22:58:40.066 INFO:teuthology.orchestra.run.smithi077.stdout:Reading state information... 2022-03-08T22:58:40.177 INFO:teuthology.orchestra.run.smithi106.stdout:Building dependency tree... 2022-03-08T22:58:40.177 INFO:teuthology.orchestra.run.smithi106.stdout:Reading state information... 2022-03-08T22:58:40.231 INFO:teuthology.orchestra.run.smithi135.stdout:ca-certificates is already the newest version (20210119~20.04.2). 2022-03-08T22:58:40.232 INFO:teuthology.orchestra.run.smithi135.stdout:curl is already the newest version (7.68.0-1ubuntu2.7). 2022-03-08T22:58:40.232 INFO:teuthology.orchestra.run.smithi135.stdout:The following NEW packages will be installed: 2022-03-08T22:58:40.233 INFO:teuthology.orchestra.run.smithi135.stdout: apt-transport-https 2022-03-08T22:58:40.248 INFO:teuthology.orchestra.run.smithi077.stdout:ca-certificates is already the newest version (20210119~20.04.2). 2022-03-08T22:58:40.248 INFO:teuthology.orchestra.run.smithi077.stdout:curl is already the newest version (7.68.0-1ubuntu2.7). 2022-03-08T22:58:40.249 INFO:teuthology.orchestra.run.smithi077.stdout:The following NEW packages will be installed: 2022-03-08T22:58:40.249 INFO:teuthology.orchestra.run.smithi077.stdout: apt-transport-https 2022-03-08T22:58:40.358 INFO:teuthology.orchestra.run.smithi106.stdout:ca-certificates is already the newest version (20210119~20.04.2). 2022-03-08T22:58:40.358 INFO:teuthology.orchestra.run.smithi106.stdout:curl is already the newest version (7.68.0-1ubuntu2.7). 2022-03-08T22:58:40.359 INFO:teuthology.orchestra.run.smithi106.stdout:The following NEW packages will be installed: 2022-03-08T22:58:40.359 INFO:teuthology.orchestra.run.smithi106.stdout: apt-transport-https 2022-03-08T22:58:40.418 INFO:teuthology.orchestra.run.smithi135.stdout:0 upgraded, 1 newly installed, 0 to remove and 85 not upgraded. 2022-03-08T22:58:40.418 INFO:teuthology.orchestra.run.smithi135.stdout:Need to get 4,680 B of archives. 2022-03-08T22:58:40.418 INFO:teuthology.orchestra.run.smithi135.stdout:After this operation, 162 kB of additional disk space will be used. 2022-03-08T22:58:40.419 INFO:teuthology.orchestra.run.smithi135.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 apt-transport-https all 2.0.6 [4,680 B] 2022-03-08T22:58:40.432 INFO:teuthology.orchestra.run.smithi077.stdout:0 upgraded, 1 newly installed, 0 to remove and 85 not upgraded. 2022-03-08T22:58:40.432 INFO:teuthology.orchestra.run.smithi077.stdout:Need to get 4,680 B of archives. 2022-03-08T22:58:40.432 INFO:teuthology.orchestra.run.smithi077.stdout:After this operation, 162 kB of additional disk space will be used. 2022-03-08T22:58:40.432 INFO:teuthology.orchestra.run.smithi077.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 apt-transport-https all 2.0.6 [4,680 B] 2022-03-08T22:58:40.546 INFO:teuthology.orchestra.run.smithi106.stdout:0 upgraded, 1 newly installed, 0 to remove and 85 not upgraded. 2022-03-08T22:58:40.546 INFO:teuthology.orchestra.run.smithi106.stdout:Need to get 4,680 B of archives. 2022-03-08T22:58:40.546 INFO:teuthology.orchestra.run.smithi106.stdout:After this operation, 162 kB of additional disk space will be used. 2022-03-08T22:58:40.547 INFO:teuthology.orchestra.run.smithi106.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 apt-transport-https all 2.0.6 [4,680 B] 2022-03-08T22:58:40.644 INFO:teuthology.orchestra.run.smithi135.stderr:debconf: unable to initialize frontend: Dialog 2022-03-08T22:58:40.645 INFO:teuthology.orchestra.run.smithi135.stderr:debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) 2022-03-08T22:58:40.645 INFO:teuthology.orchestra.run.smithi135.stderr:debconf: falling back to frontend: Readline 2022-03-08T22:58:40.667 INFO:teuthology.orchestra.run.smithi135.stderr:debconf: unable to initialize frontend: Readline 2022-03-08T22:58:40.667 INFO:teuthology.orchestra.run.smithi135.stderr:debconf: (This frontend requires a controlling tty.) 2022-03-08T22:58:40.667 INFO:teuthology.orchestra.run.smithi135.stderr:debconf: falling back to frontend: Teletype 2022-03-08T22:58:40.673 INFO:teuthology.orchestra.run.smithi135.stderr:dpkg-preconfigure: unable to re-open stdin: 2022-03-08T22:58:40.721 INFO:teuthology.orchestra.run.smithi135.stdout:Fetched 4,680 B in 0s (26.4 kB/s) 2022-03-08T22:58:40.761 INFO:teuthology.orchestra.run.smithi077.stderr:debconf: unable to initialize frontend: Dialog 2022-03-08T22:58:40.761 INFO:teuthology.orchestra.run.smithi077.stderr:debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) 2022-03-08T22:58:40.762 INFO:teuthology.orchestra.run.smithi077.stderr:debconf: falling back to frontend: Readline 2022-03-08T22:58:40.782 INFO:teuthology.orchestra.run.smithi106.stderr:debconf: unable to initialize frontend: Dialog 2022-03-08T22:58:40.782 INFO:teuthology.orchestra.run.smithi106.stderr:debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) 2022-03-08T22:58:40.783 INFO:teuthology.orchestra.run.smithi106.stderr:debconf: falling back to frontend: Readline 2022-03-08T22:58:40.788 INFO:teuthology.orchestra.run.smithi077.stderr:debconf: unable to initialize frontend: Readline 2022-03-08T22:58:40.789 INFO:teuthology.orchestra.run.smithi077.stderr:debconf: (This frontend requires a controlling tty.) 2022-03-08T22:58:40.789 INFO:teuthology.orchestra.run.smithi077.stderr:debconf: falling back to frontend: Teletype 2022-03-08T22:58:40.794 INFO:teuthology.orchestra.run.smithi077.stderr:dpkg-preconfigure: unable to re-open stdin: 2022-03-08T22:58:40.798 INFO:teuthology.orchestra.run.smithi106.stderr:debconf: unable to initialize frontend: Readline 2022-03-08T22:58:40.798 INFO:teuthology.orchestra.run.smithi106.stderr:debconf: (This frontend requires a controlling tty.) 2022-03-08T22:58:40.798 INFO:teuthology.orchestra.run.smithi106.stderr:debconf: falling back to frontend: Teletype 2022-03-08T22:58:40.803 INFO:teuthology.orchestra.run.smithi106.stderr:dpkg-preconfigure: unable to re-open stdin: 2022-03-08T22:58:40.811 INFO:teuthology.orchestra.run.smithi135.stdout:Selecting previously unselected package apt-transport-https. 2022-03-08T22:58:40.847 INFO:teuthology.orchestra.run.smithi077.stdout:Fetched 4,680 B in 0s (26.6 kB/s) 2022-03-08T22:58:40.853 INFO:teuthology.orchestra.run.smithi106.stdout:Fetched 4,680 B in 0s (26.4 kB/s) 2022-03-08T22:58:40.855 INFO:teuthology.orchestra.run.smithi135.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109240 files and directories currently installed.) 2022-03-08T22:58:40.858 INFO:teuthology.orchestra.run.smithi135.stdout:Preparing to unpack .../apt-transport-https_2.0.6_all.deb ... 2022-03-08T22:58:40.902 INFO:teuthology.orchestra.run.smithi135.stdout:Unpacking apt-transport-https (2.0.6) ... 2022-03-08T22:58:40.938 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package apt-transport-https. 2022-03-08T22:58:40.942 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package apt-transport-https. 2022-03-08T22:58:40.980 INFO:teuthology.orchestra.run.smithi077.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109240 files and directories currently installed.) 2022-03-08T22:58:40.983 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../apt-transport-https_2.0.6_all.deb ... 2022-03-08T22:58:40.985 INFO:teuthology.orchestra.run.smithi106.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109240 files and directories currently installed.) 2022-03-08T22:58:40.988 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../apt-transport-https_2.0.6_all.deb ... 2022-03-08T22:58:41.028 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking apt-transport-https (2.0.6) ... 2022-03-08T22:58:41.032 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking apt-transport-https (2.0.6) ... 2022-03-08T22:58:41.190 INFO:teuthology.orchestra.run.smithi135.stdout:Setting up apt-transport-https (2.0.6) ... 2022-03-08T22:58:41.286 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up apt-transport-https (2.0.6) ... 2022-03-08T22:58:41.305 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up apt-transport-https (2.0.6) ... 2022-03-08T22:58:41.680 INFO:teuthology.orchestra.run.smithi135.stdout:deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main 2022-03-08T22:58:41.695 INFO:teuthology.orchestra.run.smithi135.stderr: 2022-03-08T22:58:41.696 INFO:teuthology.orchestra.run.smithi135.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-08T22:58:41.696 INFO:teuthology.orchestra.run.smithi135.stderr: 2022-03-08T22:58:41.819 INFO:teuthology.orchestra.run.smithi106.stdout:deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main 2022-03-08T22:58:41.832 INFO:teuthology.orchestra.run.smithi077.stdout:deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main 2022-03-08T22:58:41.834 INFO:teuthology.orchestra.run.smithi106.stderr: 2022-03-08T22:58:41.834 INFO:teuthology.orchestra.run.smithi106.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-08T22:58:41.834 INFO:teuthology.orchestra.run.smithi106.stderr: 2022-03-08T22:58:41.847 INFO:teuthology.orchestra.run.smithi077.stderr: 2022-03-08T22:58:41.847 INFO:teuthology.orchestra.run.smithi077.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-08T22:58:41.847 INFO:teuthology.orchestra.run.smithi077.stderr: 2022-03-08T22:58:41.946 INFO:teuthology.orchestra.run.smithi135.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-08T22:58:41.947 INFO:teuthology.orchestra.run.smithi135.stdout:Hit:3 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-03-08T22:58:41.966 INFO:teuthology.orchestra.run.smithi106.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-03-08T22:58:42.016 INFO:teuthology.orchestra.run.smithi135.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-03-08T22:58:42.079 INFO:teuthology.orchestra.run.smithi106.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-08T22:58:42.081 INFO:teuthology.orchestra.run.smithi135.stdout:Get:1 https://packages.cloud.google.com/apt kubernetes-xenial InRelease [9,383 B] 2022-03-08T22:58:42.101 INFO:teuthology.orchestra.run.smithi077.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-08T22:58:42.101 INFO:teuthology.orchestra.run.smithi077.stdout:Hit:3 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-03-08T22:58:42.102 INFO:teuthology.orchestra.run.smithi135.stdout:Hit:5 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-03-08T22:58:42.161 INFO:teuthology.orchestra.run.smithi106.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-03-08T22:58:42.171 INFO:teuthology.orchestra.run.smithi077.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-03-08T22:58:42.230 INFO:teuthology.orchestra.run.smithi077.stdout:Get:1 https://packages.cloud.google.com/apt kubernetes-xenial InRelease [9,383 B] 2022-03-08T22:58:42.231 INFO:teuthology.orchestra.run.smithi106.stdout:Get:2 https://packages.cloud.google.com/apt kubernetes-xenial InRelease [9,383 B] 2022-03-08T22:58:42.245 INFO:teuthology.orchestra.run.smithi106.stdout:Hit:5 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-03-08T22:58:42.252 INFO:teuthology.orchestra.run.smithi077.stdout:Hit:5 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-03-08T22:58:42.456 INFO:teuthology.orchestra.run.smithi135.stdout:Get:6 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 Packages [53.9 kB] 2022-03-08T22:58:42.480 INFO:teuthology.orchestra.run.smithi106.stdout:Get:6 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 Packages [53.9 kB] 2022-03-08T22:58:42.605 INFO:teuthology.orchestra.run.smithi077.stdout:Get:6 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 Packages [53.9 kB] 2022-03-08T22:58:42.651 INFO:teuthology.orchestra.run.smithi135.stdout:Fetched 63.3 kB in 1s (77.8 kB/s) 2022-03-08T22:58:42.667 INFO:teuthology.orchestra.run.smithi106.stdout:Fetched 63.3 kB in 1s (92.0 kB/s) 2022-03-08T22:58:42.795 INFO:teuthology.orchestra.run.smithi077.stdout:Fetched 63.3 kB in 1s (78.8 kB/s) 2022-03-08T22:58:43.511 INFO:teuthology.orchestra.run.smithi135.stdout:Reading package lists... 2022-03-08T22:58:43.530 INFO:teuthology.orchestra.run.smithi106.stdout:Reading package lists... 2022-03-08T22:58:43.645 INFO:teuthology.orchestra.run.smithi077.stdout:Reading package lists... 2022-03-08T22:58:43.689 INFO:teuthology.orchestra.run.smithi135.stdout:Building dependency tree... 2022-03-08T22:58:43.689 INFO:teuthology.orchestra.run.smithi135.stdout:Reading state information... 2022-03-08T22:58:43.700 INFO:teuthology.orchestra.run.smithi106.stdout:Building dependency tree... 2022-03-08T22:58:43.701 INFO:teuthology.orchestra.run.smithi106.stdout:Reading state information... 2022-03-08T22:58:43.777 INFO:teuthology.orchestra.run.smithi135.stdout:85 packages can be upgraded. Run 'apt list --upgradable' to see them. 2022-03-08T22:58:43.789 INFO:teuthology.orchestra.run.smithi106.stdout:85 packages can be upgraded. Run 'apt list --upgradable' to see them. 2022-03-08T22:58:43.790 INFO:teuthology.orchestra.run.smithi135.stderr: 2022-03-08T22:58:43.790 INFO:teuthology.orchestra.run.smithi135.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-08T22:58:43.791 INFO:teuthology.orchestra.run.smithi135.stderr: 2022-03-08T22:58:43.801 INFO:teuthology.orchestra.run.smithi106.stderr: 2022-03-08T22:58:43.802 INFO:teuthology.orchestra.run.smithi106.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-08T22:58:43.802 INFO:teuthology.orchestra.run.smithi106.stderr: 2022-03-08T22:58:43.820 INFO:teuthology.orchestra.run.smithi077.stdout:Building dependency tree... 2022-03-08T22:58:43.821 INFO:teuthology.orchestra.run.smithi077.stdout:Reading state information... 2022-03-08T22:58:43.846 INFO:teuthology.orchestra.run.smithi135.stdout:Reading package lists... 2022-03-08T22:58:43.856 INFO:teuthology.orchestra.run.smithi106.stdout:Reading package lists... 2022-03-08T22:58:43.917 INFO:teuthology.orchestra.run.smithi077.stdout:85 packages can be upgraded. Run 'apt list --upgradable' to see them. 2022-03-08T22:58:43.933 INFO:teuthology.orchestra.run.smithi077.stderr: 2022-03-08T22:58:43.934 INFO:teuthology.orchestra.run.smithi077.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-08T22:58:43.934 INFO:teuthology.orchestra.run.smithi077.stderr: 2022-03-08T22:58:43.996 INFO:teuthology.orchestra.run.smithi077.stdout:Reading package lists... 2022-03-08T22:58:44.022 INFO:teuthology.orchestra.run.smithi135.stdout:Building dependency tree... 2022-03-08T22:58:44.023 INFO:teuthology.orchestra.run.smithi135.stdout:Reading state information... 2022-03-08T22:58:44.031 INFO:teuthology.orchestra.run.smithi106.stdout:Building dependency tree... 2022-03-08T22:58:44.031 INFO:teuthology.orchestra.run.smithi106.stdout:Reading state information... 2022-03-08T22:58:44.166 INFO:teuthology.orchestra.run.smithi135.stdout:bridge-utils is already the newest version (1.6-2ubuntu1). 2022-03-08T22:58:44.166 INFO:teuthology.orchestra.run.smithi135.stdout:bridge-utils set to manually installed. 2022-03-08T22:58:44.166 INFO:teuthology.orchestra.run.smithi135.stdout:The following additional packages will be installed: 2022-03-08T22:58:44.167 INFO:teuthology.orchestra.run.smithi135.stdout: conntrack cri-tools ebtables kubernetes-cni socat 2022-03-08T22:58:44.167 INFO:teuthology.orchestra.run.smithi135.stdout:Suggested packages: 2022-03-08T22:58:44.168 INFO:teuthology.orchestra.run.smithi135.stdout: nftables 2022-03-08T22:58:44.168 INFO:teuthology.orchestra.run.smithi077.stdout:Building dependency tree... 2022-03-08T22:58:44.169 INFO:teuthology.orchestra.run.smithi077.stdout:Reading state information... 2022-03-08T22:58:44.178 INFO:teuthology.orchestra.run.smithi106.stdout:bridge-utils is already the newest version (1.6-2ubuntu1). 2022-03-08T22:58:44.178 INFO:teuthology.orchestra.run.smithi106.stdout:bridge-utils set to manually installed. 2022-03-08T22:58:44.179 INFO:teuthology.orchestra.run.smithi106.stdout:The following additional packages will be installed: 2022-03-08T22:58:44.179 INFO:teuthology.orchestra.run.smithi106.stdout: conntrack cri-tools ebtables kubernetes-cni socat 2022-03-08T22:58:44.180 INFO:teuthology.orchestra.run.smithi106.stdout:Suggested packages: 2022-03-08T22:58:44.180 INFO:teuthology.orchestra.run.smithi106.stdout: nftables 2022-03-08T22:58:44.220 INFO:teuthology.orchestra.run.smithi135.stdout:The following NEW packages will be installed: 2022-03-08T22:58:44.221 INFO:teuthology.orchestra.run.smithi135.stdout: conntrack cri-tools ebtables kubeadm kubectl kubelet kubernetes-cni socat 2022-03-08T22:58:44.227 INFO:teuthology.orchestra.run.smithi106.stdout:The following NEW packages will be installed: 2022-03-08T22:58:44.230 INFO:teuthology.orchestra.run.smithi106.stdout: conntrack cri-tools ebtables kubeadm kubectl kubelet kubernetes-cni socat 2022-03-08T22:58:44.307 INFO:teuthology.orchestra.run.smithi077.stdout:bridge-utils is already the newest version (1.6-2ubuntu1). 2022-03-08T22:58:44.308 INFO:teuthology.orchestra.run.smithi077.stdout:bridge-utils set to manually installed. 2022-03-08T22:58:44.308 INFO:teuthology.orchestra.run.smithi077.stdout:The following additional packages will be installed: 2022-03-08T22:58:44.309 INFO:teuthology.orchestra.run.smithi077.stdout: conntrack cri-tools ebtables kubernetes-cni socat 2022-03-08T22:58:44.309 INFO:teuthology.orchestra.run.smithi077.stdout:Suggested packages: 2022-03-08T22:58:44.309 INFO:teuthology.orchestra.run.smithi077.stdout: nftables 2022-03-08T22:58:44.352 INFO:teuthology.orchestra.run.smithi077.stdout:The following NEW packages will be installed: 2022-03-08T22:58:44.353 INFO:teuthology.orchestra.run.smithi077.stdout: conntrack cri-tools ebtables kubeadm kubectl kubelet kubernetes-cni socat 2022-03-08T22:58:44.414 INFO:teuthology.orchestra.run.smithi135.stdout:0 upgraded, 8 newly installed, 0 to remove and 85 not upgraded. 2022-03-08T22:58:44.414 INFO:teuthology.orchestra.run.smithi135.stdout:Need to get 73.6 MB of archives. 2022-03-08T22:58:44.415 INFO:teuthology.orchestra.run.smithi135.stdout:After this operation, 321 MB of additional disk space will be used. 2022-03-08T22:58:44.415 INFO:teuthology.orchestra.run.smithi135.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 conntrack amd64 1:1.4.5-2 [30.3 kB] 2022-03-08T22:58:44.420 INFO:teuthology.orchestra.run.smithi106.stdout:0 upgraded, 8 newly installed, 0 to remove and 85 not upgraded. 2022-03-08T22:58:44.421 INFO:teuthology.orchestra.run.smithi106.stdout:Need to get 73.6 MB of archives. 2022-03-08T22:58:44.421 INFO:teuthology.orchestra.run.smithi106.stdout:After this operation, 321 MB of additional disk space will be used. 2022-03-08T22:58:44.421 INFO:teuthology.orchestra.run.smithi106.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 conntrack amd64 1:1.4.5-2 [30.3 kB] 2022-03-08T22:58:44.453 INFO:teuthology.orchestra.run.smithi135.stdout:Get:1 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 cri-tools amd64 1.19.0-00 [11.2 MB] 2022-03-08T22:58:44.524 INFO:teuthology.orchestra.run.smithi106.stdout:Get:1 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 cri-tools amd64 1.19.0-00 [11.2 MB] 2022-03-08T22:58:44.552 INFO:teuthology.orchestra.run.smithi077.stdout:0 upgraded, 8 newly installed, 0 to remove and 85 not upgraded. 2022-03-08T22:58:44.553 INFO:teuthology.orchestra.run.smithi077.stdout:Need to get 73.6 MB of archives. 2022-03-08T22:58:44.553 INFO:teuthology.orchestra.run.smithi077.stdout:After this operation, 321 MB of additional disk space will be used. 2022-03-08T22:58:44.553 INFO:teuthology.orchestra.run.smithi077.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 conntrack amd64 1:1.4.5-2 [30.3 kB] 2022-03-08T22:58:44.591 INFO:teuthology.orchestra.run.smithi106.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/main amd64 ebtables amd64 2.0.11-3build1 [80.3 kB] 2022-03-08T22:58:44.594 INFO:teuthology.orchestra.run.smithi135.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/main amd64 ebtables amd64 2.0.11-3build1 [80.3 kB] 2022-03-08T22:58:44.601 INFO:teuthology.orchestra.run.smithi077.stdout:Get:1 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 cri-tools amd64 1.19.0-00 [11.2 MB] 2022-03-08T22:58:44.723 INFO:teuthology.orchestra.run.smithi106.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2022-03-08T22:58:44.750 INFO:teuthology.orchestra.run.smithi135.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2022-03-08T22:58:44.762 INFO:teuthology.orchestra.run.smithi077.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/main amd64 ebtables amd64 2.0.11-3build1 [80.3 kB] 2022-03-08T22:58:44.811 INFO:teuthology.orchestra.run.smithi135.stdout:Get:2 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubernetes-cni amd64 0.8.7-00 [25.0 MB] 2022-03-08T22:58:44.876 INFO:teuthology.orchestra.run.smithi077.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2022-03-08T22:58:44.877 INFO:teuthology.orchestra.run.smithi106.stdout:Get:2 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubernetes-cni amd64 0.8.7-00 [25.0 MB] 2022-03-08T22:58:45.443 INFO:teuthology.orchestra.run.smithi135.stdout:Get:3 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubelet amd64 1.23.4-00 [19.5 MB] 2022-03-08T22:58:45.683 INFO:teuthology.orchestra.run.smithi106.stdout:Get:3 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubelet amd64 1.23.4-00 [19.5 MB] 2022-03-08T22:58:46.023 INFO:teuthology.orchestra.run.smithi135.stdout:Get:5 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubectl amd64 1.23.4-00 [8,927 kB] 2022-03-08T22:58:46.124 INFO:teuthology.orchestra.run.smithi135.stdout:Get:6 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubeadm amd64 1.23.4-00 [8,583 kB] 2022-03-08T22:58:46.191 INFO:teuthology.orchestra.run.smithi106.stdout:Get:5 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubectl amd64 1.23.4-00 [8,927 kB] 2022-03-08T22:58:46.361 INFO:teuthology.orchestra.run.smithi106.stdout:Get:6 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubeadm amd64 1.23.4-00 [8,583 kB] 2022-03-08T22:58:46.574 INFO:teuthology.orchestra.run.smithi135.stderr:debconf: unable to initialize frontend: Dialog 2022-03-08T22:58:46.574 INFO:teuthology.orchestra.run.smithi135.stderr:debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) 2022-03-08T22:58:46.574 INFO:teuthology.orchestra.run.smithi135.stderr:debconf: falling back to frontend: Readline 2022-03-08T22:58:46.583 INFO:teuthology.orchestra.run.smithi135.stderr:debconf: unable to initialize frontend: Readline 2022-03-08T22:58:46.583 INFO:teuthology.orchestra.run.smithi135.stderr:debconf: (This frontend requires a controlling tty.) 2022-03-08T22:58:46.584 INFO:teuthology.orchestra.run.smithi135.stderr:debconf: falling back to frontend: Teletype 2022-03-08T22:58:46.588 INFO:teuthology.orchestra.run.smithi135.stderr:dpkg-preconfigure: unable to re-open stdin: 2022-03-08T22:58:46.637 INFO:teuthology.orchestra.run.smithi135.stdout:Fetched 73.6 MB in 2s (34.6 MB/s) 2022-03-08T22:58:46.662 INFO:teuthology.orchestra.run.smithi077.stdout:Get:2 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubernetes-cni amd64 0.8.7-00 [25.0 MB] 2022-03-08T22:58:46.710 INFO:teuthology.orchestra.run.smithi106.stderr:debconf: unable to initialize frontend: Dialog 2022-03-08T22:58:46.711 INFO:teuthology.orchestra.run.smithi106.stderr:debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) 2022-03-08T22:58:46.711 INFO:teuthology.orchestra.run.smithi106.stderr:debconf: falling back to frontend: Readline 2022-03-08T22:58:46.719 INFO:teuthology.orchestra.run.smithi106.stderr:debconf: unable to initialize frontend: Readline 2022-03-08T22:58:46.719 INFO:teuthology.orchestra.run.smithi106.stderr:debconf: (This frontend requires a controlling tty.) 2022-03-08T22:58:46.719 INFO:teuthology.orchestra.run.smithi106.stderr:debconf: falling back to frontend: Teletype 2022-03-08T22:58:46.725 INFO:teuthology.orchestra.run.smithi106.stderr:dpkg-preconfigure: unable to re-open stdin: 2022-03-08T22:58:46.726 INFO:teuthology.orchestra.run.smithi135.stdout:Selecting previously unselected package conntrack. 2022-03-08T22:58:46.770 INFO:teuthology.orchestra.run.smithi135.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109244 files and directories currently installed.) 2022-03-08T22:58:46.774 INFO:teuthology.orchestra.run.smithi135.stdout:Preparing to unpack .../0-conntrack_1%3a1.4.5-2_amd64.deb ... 2022-03-08T22:58:46.776 INFO:teuthology.orchestra.run.smithi106.stdout:Fetched 73.6 MB in 2s (32.6 MB/s) 2022-03-08T22:58:46.816 INFO:teuthology.orchestra.run.smithi135.stdout:Unpacking conntrack (1:1.4.5-2) ... 2022-03-08T22:58:46.864 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package conntrack. 2022-03-08T22:58:46.908 INFO:teuthology.orchestra.run.smithi106.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109244 files and directories currently installed.) 2022-03-08T22:58:46.911 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../0-conntrack_1%3a1.4.5-2_amd64.deb ... 2022-03-08T22:58:46.955 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking conntrack (1:1.4.5-2) ... 2022-03-08T22:58:47.169 INFO:teuthology.orchestra.run.smithi135.stdout:Selecting previously unselected package cri-tools. 2022-03-08T22:58:47.182 INFO:teuthology.orchestra.run.smithi135.stdout:Preparing to unpack .../1-cri-tools_1.19.0-00_amd64.deb ... 2022-03-08T22:58:47.219 INFO:teuthology.orchestra.run.smithi135.stdout:Unpacking cri-tools (1.19.0-00) ... 2022-03-08T22:58:47.232 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package cri-tools. 2022-03-08T22:58:47.246 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../1-cri-tools_1.19.0-00_amd64.deb ... 2022-03-08T22:58:47.316 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking cri-tools (1.19.0-00) ... 2022-03-08T22:58:47.572 INFO:teuthology.orchestra.run.smithi077.stdout:Get:3 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubelet amd64 1.23.4-00 [19.5 MB] 2022-03-08T22:58:47.907 INFO:teuthology.orchestra.run.smithi077.stdout:Get:5 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubectl amd64 1.23.4-00 [8,927 kB] 2022-03-08T22:58:48.108 INFO:teuthology.orchestra.run.smithi077.stdout:Get:6 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubeadm amd64 1.23.4-00 [8,583 kB] 2022-03-08T22:58:48.289 INFO:teuthology.orchestra.run.smithi135.stdout:Selecting previously unselected package ebtables. 2022-03-08T22:58:48.305 INFO:teuthology.orchestra.run.smithi135.stdout:Preparing to unpack .../2-ebtables_2.0.11-3build1_amd64.deb ... 2022-03-08T22:58:48.364 INFO:teuthology.orchestra.run.smithi135.stdout:Unpacking ebtables (2.0.11-3build1) ... 2022-03-08T22:58:48.368 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package ebtables. 2022-03-08T22:58:48.381 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../2-ebtables_2.0.11-3build1_amd64.deb ... 2022-03-08T22:58:48.434 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking ebtables (2.0.11-3build1) ... 2022-03-08T22:58:48.538 INFO:teuthology.orchestra.run.smithi077.stderr:debconf: unable to initialize frontend: Dialog 2022-03-08T22:58:48.539 INFO:teuthology.orchestra.run.smithi077.stderr:debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) 2022-03-08T22:58:48.539 INFO:teuthology.orchestra.run.smithi077.stderr:debconf: falling back to frontend: Readline 2022-03-08T22:58:48.547 INFO:teuthology.orchestra.run.smithi077.stderr:debconf: unable to initialize frontend: Readline 2022-03-08T22:58:48.547 INFO:teuthology.orchestra.run.smithi077.stderr:debconf: (This frontend requires a controlling tty.) 2022-03-08T22:58:48.548 INFO:teuthology.orchestra.run.smithi077.stderr:debconf: falling back to frontend: Teletype 2022-03-08T22:58:48.553 INFO:teuthology.orchestra.run.smithi077.stderr:dpkg-preconfigure: unable to re-open stdin: 2022-03-08T22:58:48.600 INFO:teuthology.orchestra.run.smithi077.stdout:Fetched 73.6 MB in 4s (18.6 MB/s) 2022-03-08T22:58:48.608 INFO:teuthology.orchestra.run.smithi135.stdout:Selecting previously unselected package kubernetes-cni. 2022-03-08T22:58:48.621 INFO:teuthology.orchestra.run.smithi135.stdout:Preparing to unpack .../3-kubernetes-cni_0.8.7-00_amd64.deb ... 2022-03-08T22:58:48.658 INFO:teuthology.orchestra.run.smithi135.stdout:Unpacking kubernetes-cni (0.8.7-00) ... 2022-03-08T22:58:48.685 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package conntrack. 2022-03-08T22:58:48.731 INFO:teuthology.orchestra.run.smithi077.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109244 files and directories currently installed.) 2022-03-08T22:58:48.734 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../0-conntrack_1%3a1.4.5-2_amd64.deb ... 2022-03-08T22:58:48.737 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package kubernetes-cni. 2022-03-08T22:58:48.753 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../3-kubernetes-cni_0.8.7-00_amd64.deb ... 2022-03-08T22:58:48.796 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking kubernetes-cni (0.8.7-00) ... 2022-03-08T22:58:48.800 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking conntrack (1:1.4.5-2) ... 2022-03-08T22:58:49.103 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package cri-tools. 2022-03-08T22:58:49.117 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../1-cri-tools_1.19.0-00_amd64.deb ... 2022-03-08T22:58:49.162 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking cri-tools (1.19.0-00) ... 2022-03-08T22:58:50.077 INFO:teuthology.orchestra.run.smithi135.stdout:Selecting previously unselected package socat. 2022-03-08T22:58:50.089 INFO:teuthology.orchestra.run.smithi135.stdout:Preparing to unpack .../4-socat_1.7.3.3-2_amd64.deb ... 2022-03-08T22:58:50.127 INFO:teuthology.orchestra.run.smithi135.stdout:Unpacking socat (1.7.3.3-2) ... 2022-03-08T22:58:50.232 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package socat. 2022-03-08T22:58:50.245 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../4-socat_1.7.3.3-2_amd64.deb ... 2022-03-08T22:58:50.256 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package ebtables. 2022-03-08T22:58:50.270 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../2-ebtables_2.0.11-3build1_amd64.deb ... 2022-03-08T22:58:50.290 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking socat (1.7.3.3-2) ... 2022-03-08T22:58:50.330 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking ebtables (2.0.11-3build1) ... 2022-03-08T22:58:50.521 INFO:teuthology.orchestra.run.smithi135.stdout:Selecting previously unselected package kubelet. 2022-03-08T22:58:50.535 INFO:teuthology.orchestra.run.smithi135.stdout:Preparing to unpack .../5-kubelet_1.23.4-00_amd64.deb ... 2022-03-08T22:58:50.580 INFO:teuthology.orchestra.run.smithi135.stdout:Unpacking kubelet (1.23.4-00) ... 2022-03-08T22:58:50.592 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package kubernetes-cni. 2022-03-08T22:58:50.607 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../3-kubernetes-cni_0.8.7-00_amd64.deb ... 2022-03-08T22:58:50.651 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking kubernetes-cni (0.8.7-00) ... 2022-03-08T22:58:50.706 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package kubelet. 2022-03-08T22:58:50.720 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../5-kubelet_1.23.4-00_amd64.deb ... 2022-03-08T22:58:50.764 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking kubelet (1.23.4-00) ... 2022-03-08T22:58:52.086 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package socat. 2022-03-08T22:58:52.099 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../4-socat_1.7.3.3-2_amd64.deb ... 2022-03-08T22:58:52.136 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking socat (1.7.3.3-2) ... 2022-03-08T22:58:52.507 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package kubelet. 2022-03-08T22:58:52.520 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../5-kubelet_1.23.4-00_amd64.deb ... 2022-03-08T22:58:52.557 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking kubelet (1.23.4-00) ... 2022-03-08T22:58:53.110 INFO:teuthology.orchestra.run.smithi135.stdout:Selecting previously unselected package kubectl. 2022-03-08T22:58:53.124 INFO:teuthology.orchestra.run.smithi135.stdout:Preparing to unpack .../6-kubectl_1.23.4-00_amd64.deb ... 2022-03-08T22:58:53.168 INFO:teuthology.orchestra.run.smithi135.stdout:Unpacking kubectl (1.23.4-00) ... 2022-03-08T22:58:53.191 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package kubectl. 2022-03-08T22:58:53.206 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../6-kubectl_1.23.4-00_amd64.deb ... 2022-03-08T22:58:53.275 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking kubectl (1.23.4-00) ... 2022-03-08T22:58:54.340 INFO:teuthology.orchestra.run.smithi135.stdout:Selecting previously unselected package kubeadm. 2022-03-08T22:58:54.354 INFO:teuthology.orchestra.run.smithi135.stdout:Preparing to unpack .../7-kubeadm_1.23.4-00_amd64.deb ... 2022-03-08T22:58:54.391 INFO:teuthology.orchestra.run.smithi135.stdout:Unpacking kubeadm (1.23.4-00) ... 2022-03-08T22:58:54.469 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package kubeadm. 2022-03-08T22:58:54.483 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../7-kubeadm_1.23.4-00_amd64.deb ... 2022-03-08T22:58:54.546 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking kubeadm (1.23.4-00) ... 2022-03-08T22:58:55.018 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package kubectl. 2022-03-08T22:58:55.025 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../6-kubectl_1.23.4-00_amd64.deb ... 2022-03-08T22:58:55.068 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking kubectl (1.23.4-00) ... 2022-03-08T22:58:55.569 INFO:teuthology.orchestra.run.smithi135.stdout:Setting up conntrack (1:1.4.5-2) ... 2022-03-08T22:58:55.687 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up conntrack (1:1.4.5-2) ... 2022-03-08T22:58:55.698 INFO:teuthology.orchestra.run.smithi135.stdout:Setting up kubectl (1.23.4-00) ... 2022-03-08T22:58:55.800 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up kubectl (1.23.4-00) ... 2022-03-08T22:58:55.843 INFO:teuthology.orchestra.run.smithi135.stdout:Setting up ebtables (2.0.11-3build1) ... 2022-03-08T22:58:55.926 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up ebtables (2.0.11-3build1) ... 2022-03-08T22:58:56.205 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package kubeadm. 2022-03-08T22:58:56.218 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../7-kubeadm_1.23.4-00_amd64.deb ... 2022-03-08T22:58:56.256 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking kubeadm (1.23.4-00) ... 2022-03-08T22:58:56.758 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up socat (1.7.3.3-2) ... 2022-03-08T22:58:56.859 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up cri-tools (1.19.0-00) ... 2022-03-08T22:58:56.977 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up kubernetes-cni (0.8.7-00) ... 2022-03-08T22:58:57.048 INFO:teuthology.orchestra.run.smithi135.stdout:Setting up socat (1.7.3.3-2) ... 2022-03-08T22:58:57.103 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up kubelet (1.23.4-00) ... 2022-03-08T22:58:57.308 INFO:teuthology.orchestra.run.smithi135.stdout:Setting up cri-tools (1.19.0-00) ... 2022-03-08T22:58:57.350 INFO:teuthology.orchestra.run.smithi106.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/kubelet.service -> /lib/systemd/system/kubelet.service. 2022-03-08T22:58:57.393 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up conntrack (1:1.4.5-2) ... 2022-03-08T22:58:57.409 INFO:teuthology.orchestra.run.smithi135.stdout:Setting up kubernetes-cni (0.8.7-00) ... 2022-03-08T22:58:57.518 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up kubectl (1.23.4-00) ... 2022-03-08T22:58:57.535 INFO:teuthology.orchestra.run.smithi135.stdout:Setting up kubelet (1.23.4-00) ... 2022-03-08T22:58:57.637 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up ebtables (2.0.11-3build1) ... 2022-03-08T22:58:57.785 INFO:teuthology.orchestra.run.smithi135.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/kubelet.service -> /lib/systemd/system/kubelet.service. 2022-03-08T22:58:57.862 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up kubeadm (1.23.4-00) ... 2022-03-08T22:58:58.254 INFO:teuthology.orchestra.run.smithi135.stdout:Setting up kubeadm (1.23.4-00) ... 2022-03-08T22:58:58.423 INFO:teuthology.orchestra.run.smithi106.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-03-08T22:58:58.492 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up socat (1.7.3.3-2) ... 2022-03-08T22:58:58.618 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up cri-tools (1.19.0-00) ... 2022-03-08T22:58:58.639 INFO:teuthology.orchestra.run.smithi135.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-03-08T22:58:58.744 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up kubernetes-cni (0.8.7-00) ... 2022-03-08T22:58:58.870 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up kubelet (1.23.4-00) ... 2022-03-08T22:58:59.093 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/kubelet.service -> /lib/systemd/system/kubelet.service. 2022-03-08T22:58:59.571 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up kubeadm (1.23.4-00) ... 2022-03-08T22:58:59.948 INFO:teuthology.orchestra.run.smithi077.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-03-08T22:59:00.812 DEBUG:teuthology.orchestra.run.smithi077:> sudo systemctl enable --now kubelet && sudo kubeadm config images pull 2022-03-08T22:59:00.857 DEBUG:teuthology.orchestra.run.smithi106:> sudo systemctl enable --now kubelet && sudo kubeadm config images pull 2022-03-08T22:59:00.860 DEBUG:teuthology.orchestra.run.smithi135:> sudo systemctl enable --now kubelet && sudo kubeadm config images pull 2022-03-08T22:59:12.745 INFO:teuthology.orchestra.run.smithi106.stdout:[config/images] Pulled k8s.gcr.io/kube-apiserver:v1.23.4 2022-03-08T22:59:13.234 INFO:teuthology.orchestra.run.smithi135.stdout:[config/images] Pulled k8s.gcr.io/kube-apiserver:v1.23.4 2022-03-08T22:59:14.382 INFO:teuthology.orchestra.run.smithi077.stdout:[config/images] Pulled k8s.gcr.io/kube-apiserver:v1.23.4 2022-03-08T22:59:15.839 INFO:teuthology.orchestra.run.smithi106.stdout:[config/images] Pulled k8s.gcr.io/kube-controller-manager:v1.23.4 2022-03-08T22:59:16.311 INFO:teuthology.orchestra.run.smithi135.stdout:[config/images] Pulled k8s.gcr.io/kube-controller-manager:v1.23.4 2022-03-08T22:59:17.309 INFO:teuthology.orchestra.run.smithi077.stdout:[config/images] Pulled k8s.gcr.io/kube-controller-manager:v1.23.4 2022-03-08T22:59:18.611 INFO:teuthology.orchestra.run.smithi106.stdout:[config/images] Pulled k8s.gcr.io/kube-scheduler:v1.23.4 2022-03-08T22:59:18.792 INFO:teuthology.orchestra.run.smithi135.stdout:[config/images] Pulled k8s.gcr.io/kube-scheduler:v1.23.4 2022-03-08T22:59:19.531 INFO:teuthology.orchestra.run.smithi077.stdout:[config/images] Pulled k8s.gcr.io/kube-scheduler:v1.23.4 2022-03-08T22:59:22.194 INFO:teuthology.orchestra.run.smithi106.stdout:[config/images] Pulled k8s.gcr.io/kube-proxy:v1.23.4 2022-03-08T22:59:22.408 INFO:teuthology.orchestra.run.smithi135.stdout:[config/images] Pulled k8s.gcr.io/kube-proxy:v1.23.4 2022-03-08T22:59:23.464 INFO:teuthology.orchestra.run.smithi077.stdout:[config/images] Pulled k8s.gcr.io/kube-proxy:v1.23.4 2022-03-08T22:59:23.694 INFO:teuthology.orchestra.run.smithi106.stdout:[config/images] Pulled k8s.gcr.io/pause:3.6 2022-03-08T22:59:23.983 INFO:teuthology.orchestra.run.smithi135.stdout:[config/images] Pulled k8s.gcr.io/pause:3.6 2022-03-08T22:59:25.014 INFO:teuthology.orchestra.run.smithi077.stdout:[config/images] Pulled k8s.gcr.io/pause:3.6 2022-03-08T22:59:30.549 INFO:teuthology.orchestra.run.smithi135.stdout:[config/images] Pulled k8s.gcr.io/etcd:3.5.1-0 2022-03-08T22:59:31.476 INFO:teuthology.orchestra.run.smithi106.stdout:[config/images] Pulled k8s.gcr.io/etcd:3.5.1-0 2022-03-08T22:59:32.338 INFO:teuthology.orchestra.run.smithi077.stdout:[config/images] Pulled k8s.gcr.io/etcd:3.5.1-0 2022-03-08T22:59:32.856 INFO:teuthology.orchestra.run.smithi135.stdout:[config/images] Pulled k8s.gcr.io/coredns/coredns:v1.8.6 2022-03-08T22:59:33.467 INFO:teuthology.orchestra.run.smithi106.stdout:[config/images] Pulled k8s.gcr.io/coredns/coredns:v1.8.6 2022-03-08T22:59:34.753 INFO:teuthology.orchestra.run.smithi077.stdout:[config/images] Pulled k8s.gcr.io/coredns/coredns:v1.8.6 2022-03-08T22:59:34.758 INFO:tasks.kubeadm:Token: abcdef.578i67scap8804og 2022-03-08T22:59:34.759 INFO:tasks.kubeadm:Remotes: {Remote(name='ubuntu@smithi077.front.sepia.ceph.com'): '172.21.15.77', Remote(name='ubuntu@smithi106.front.sepia.ceph.com'): '172.21.15.106', Remote(name='ubuntu@smithi135.front.sepia.ceph.com'): '172.21.15.135'} 2022-03-08T22:59:34.760 DEBUG:teuthology.orchestra.run.smithi077:> sudo kubeadm init --node-name smithi077 --token abcdef.578i67scap8804og --pod-network-cidr 10.250.96.0/21 2022-03-08T22:59:34.997 INFO:teuthology.orchestra.run.smithi077.stdout:[init] Using Kubernetes version: v1.23.4 2022-03-08T22:59:34.997 INFO:teuthology.orchestra.run.smithi077.stdout:[preflight] Running pre-flight checks 2022-03-08T22:59:35.391 INFO:teuthology.orchestra.run.smithi077.stdout:[preflight] Pulling images required for setting up a Kubernetes cluster 2022-03-08T22:59:35.391 INFO:teuthology.orchestra.run.smithi077.stdout:[preflight] This might take a minute or two, depending on the speed of your internet connection 2022-03-08T22:59:35.391 INFO:teuthology.orchestra.run.smithi077.stdout:[preflight] You can also perform this action in beforehand using 'kubeadm config images pull' 2022-03-08T22:59:35.690 INFO:teuthology.orchestra.run.smithi077.stdout:[certs] Using certificateDir folder "/etc/kubernetes/pki" 2022-03-08T22:59:35.786 INFO:teuthology.orchestra.run.smithi077.stdout:[certs] Generating "ca" certificate and key 2022-03-08T22:59:35.863 INFO:teuthology.orchestra.run.smithi077.stdout:[certs] Generating "apiserver" certificate and key 2022-03-08T22:59:35.863 INFO:teuthology.orchestra.run.smithi077.stdout:[certs] apiserver serving cert is signed for DNS names [kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local smithi077] and IPs [10.96.0.1 172.21.15.77] 2022-03-08T22:59:35.925 INFO:teuthology.orchestra.run.smithi077.stdout:[certs] Generating "apiserver-kubelet-client" certificate and key 2022-03-08T22:59:36.089 INFO:teuthology.orchestra.run.smithi077.stdout:[certs] Generating "front-proxy-ca" certificate and key 2022-03-08T22:59:36.309 INFO:teuthology.orchestra.run.smithi077.stdout:[certs] Generating "front-proxy-client" certificate and key 2022-03-08T22:59:36.500 INFO:teuthology.orchestra.run.smithi077.stdout:[certs] Generating "etcd/ca" certificate and key 2022-03-08T22:59:36.591 INFO:teuthology.orchestra.run.smithi077.stdout:[certs] Generating "etcd/server" certificate and key 2022-03-08T22:59:36.591 INFO:teuthology.orchestra.run.smithi077.stdout:[certs] etcd/server serving cert is signed for DNS names [localhost smithi077] and IPs [172.21.15.77 127.0.0.1 ::1] 2022-03-08T22:59:36.842 INFO:teuthology.orchestra.run.smithi077.stdout:[certs] Generating "etcd/peer" certificate and key 2022-03-08T22:59:36.842 INFO:teuthology.orchestra.run.smithi077.stdout:[certs] etcd/peer serving cert is signed for DNS names [localhost smithi077] and IPs [172.21.15.77 127.0.0.1 ::1] 2022-03-08T22:59:37.208 INFO:teuthology.orchestra.run.smithi077.stdout:[certs] Generating "etcd/healthcheck-client" certificate and key 2022-03-08T22:59:37.313 INFO:teuthology.orchestra.run.smithi077.stdout:[certs] Generating "apiserver-etcd-client" certificate and key 2022-03-08T22:59:37.397 INFO:teuthology.orchestra.run.smithi077.stdout:[certs] Generating "sa" key and public key 2022-03-08T22:59:37.398 INFO:teuthology.orchestra.run.smithi077.stdout:[kubeconfig] Using kubeconfig folder "/etc/kubernetes" 2022-03-08T22:59:37.589 INFO:teuthology.orchestra.run.smithi077.stdout:[kubeconfig] Writing "admin.conf" kubeconfig file 2022-03-08T22:59:37.728 INFO:teuthology.orchestra.run.smithi077.stdout:[kubeconfig] Writing "kubelet.conf" kubeconfig file 2022-03-08T22:59:38.024 INFO:teuthology.orchestra.run.smithi077.stdout:[kubeconfig] Writing "controller-manager.conf" kubeconfig file 2022-03-08T22:59:38.126 INFO:teuthology.orchestra.run.smithi077.stdout:[kubeconfig] Writing "scheduler.conf" kubeconfig file 2022-03-08T22:59:38.139 INFO:teuthology.orchestra.run.smithi077.stdout:[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" 2022-03-08T22:59:38.140 INFO:teuthology.orchestra.run.smithi077.stdout:[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" 2022-03-08T22:59:38.141 INFO:teuthology.orchestra.run.smithi077.stdout:[kubelet-start] Starting the kubelet 2022-03-08T22:59:38.334 INFO:teuthology.orchestra.run.smithi077.stdout:[control-plane] Using manifest folder "/etc/kubernetes/manifests" 2022-03-08T22:59:38.335 INFO:teuthology.orchestra.run.smithi077.stdout:[control-plane] Creating static Pod manifest for "kube-apiserver" 2022-03-08T22:59:38.338 INFO:teuthology.orchestra.run.smithi077.stdout:[control-plane] Creating static Pod manifest for "kube-controller-manager" 2022-03-08T22:59:38.340 INFO:teuthology.orchestra.run.smithi077.stdout:[control-plane] Creating static Pod manifest for "kube-scheduler" 2022-03-08T22:59:38.342 INFO:teuthology.orchestra.run.smithi077.stdout:[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests" 2022-03-08T22:59:38.343 INFO:teuthology.orchestra.run.smithi077.stdout:[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s 2022-03-08T22:59:54.346 INFO:teuthology.orchestra.run.smithi077.stdout:[apiclient] All control plane components are healthy after 16.002539 seconds 2022-03-08T22:59:54.347 INFO:teuthology.orchestra.run.smithi077.stdout:[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace 2022-03-08T22:59:54.361 INFO:teuthology.orchestra.run.smithi077.stdout:[kubelet] Creating a ConfigMap "kubelet-config-1.23" in namespace kube-system with the configuration for the kubelets in the cluster 2022-03-08T22:59:54.362 INFO:teuthology.orchestra.run.smithi077.stdout:NOTE: The "kubelet-config-1.23" naming of the kubelet ConfigMap is deprecated. Once the UnversionedKubeletConfigMap feature gate graduates to Beta the default name will become just "kubelet-config". Kubeadm upgrade will handle this transition transparently. 2022-03-08T22:59:54.911 INFO:teuthology.orchestra.run.smithi077.stdout:[upload-certs] Skipping phase. Please see --upload-certs 2022-03-08T22:59:54.912 INFO:teuthology.orchestra.run.smithi077.stdout:[mark-control-plane] Marking the node smithi077 as control-plane by adding the labels: [node-role.kubernetes.io/master(deprecated) node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers] 2022-03-08T22:59:54.912 INFO:teuthology.orchestra.run.smithi077.stdout:[mark-control-plane] Marking the node smithi077 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule] 2022-03-08T22:59:55.460 INFO:teuthology.orchestra.run.smithi077.stdout:[bootstrap-token] Using token: abcdef.578i67scap8804og 2022-03-08T22:59:55.461 INFO:teuthology.orchestra.run.smithi077.stdout:[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles 2022-03-08T22:59:55.478 INFO:teuthology.orchestra.run.smithi077.stdout:[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes 2022-03-08T22:59:55.495 INFO:teuthology.orchestra.run.smithi077.stdout:[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials 2022-03-08T22:59:55.503 INFO:teuthology.orchestra.run.smithi077.stdout:[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token 2022-03-08T22:59:55.511 INFO:teuthology.orchestra.run.smithi077.stdout:[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster 2022-03-08T22:59:55.533 INFO:teuthology.orchestra.run.smithi077.stdout:[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace 2022-03-08T22:59:55.628 INFO:teuthology.orchestra.run.smithi077.stdout:[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key 2022-03-08T22:59:56.079 INFO:teuthology.orchestra.run.smithi077.stdout:[addons] Applied essential addon: CoreDNS 2022-03-08T22:59:56.249 INFO:teuthology.orchestra.run.smithi077.stdout:[addons] Applied essential addon: kube-proxy 2022-03-08T22:59:56.250 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-03-08T22:59:56.250 INFO:teuthology.orchestra.run.smithi077.stdout:Your Kubernetes control-plane has initialized successfully! 2022-03-08T22:59:56.250 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-03-08T22:59:56.251 INFO:teuthology.orchestra.run.smithi077.stdout:To start using your cluster, you need to run the following as a regular user: 2022-03-08T22:59:56.251 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-03-08T22:59:56.251 INFO:teuthology.orchestra.run.smithi077.stdout: mkdir -p $HOME/.kube 2022-03-08T22:59:56.251 INFO:teuthology.orchestra.run.smithi077.stdout: sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config 2022-03-08T22:59:56.252 INFO:teuthology.orchestra.run.smithi077.stdout: sudo chown $(id -u):$(id -g) $HOME/.kube/config 2022-03-08T22:59:56.252 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-03-08T22:59:56.252 INFO:teuthology.orchestra.run.smithi077.stdout:Alternatively, if you are the root user, you can run: 2022-03-08T22:59:56.252 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-03-08T22:59:56.253 INFO:teuthology.orchestra.run.smithi077.stdout: export KUBECONFIG=/etc/kubernetes/admin.conf 2022-03-08T22:59:56.253 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-03-08T22:59:56.253 INFO:teuthology.orchestra.run.smithi077.stdout:You should now deploy a pod network to the cluster. 2022-03-08T22:59:56.253 INFO:teuthology.orchestra.run.smithi077.stdout:Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at: 2022-03-08T22:59:56.253 INFO:teuthology.orchestra.run.smithi077.stdout: https://kubernetes.io/docs/concepts/cluster-administration/addons/ 2022-03-08T22:59:56.254 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-03-08T22:59:56.254 INFO:teuthology.orchestra.run.smithi077.stdout:Then you can join any number of worker nodes by running the following on each as root: 2022-03-08T22:59:56.254 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-03-08T22:59:56.254 INFO:teuthology.orchestra.run.smithi077.stdout:kubeadm join 172.21.15.77:6443 --token abcdef.578i67scap8804og \ 2022-03-08T22:59:56.255 INFO:teuthology.orchestra.run.smithi077.stdout: --discovery-token-ca-cert-hash sha256:4d01fc3dd935a478abba5ecdb79c17d60625968d1de0e901f8c74accb9026fff 2022-03-08T22:59:56.256 DEBUG:teuthology.orchestra.run.smithi106:> sudo kubeadm join 172.21.15.77:6443 --node-name smithi106 --token abcdef.578i67scap8804og --discovery-token-unsafe-skip-ca-verification 2022-03-08T22:59:56.259 DEBUG:teuthology.orchestra.run.smithi135:> sudo kubeadm join 172.21.15.77:6443 --node-name smithi135 --token abcdef.578i67scap8804og --discovery-token-unsafe-skip-ca-verification 2022-03-08T22:59:56.295 INFO:teuthology.orchestra.run.smithi106.stdout:[preflight] Running pre-flight checks 2022-03-08T22:59:56.304 INFO:teuthology.orchestra.run.smithi135.stdout:[preflight] Running pre-flight checks 2022-03-08T23:00:09.103 INFO:teuthology.orchestra.run.smithi106.stdout:[preflight] Reading configuration from the cluster... 2022-03-08T23:00:09.104 INFO:teuthology.orchestra.run.smithi106.stdout:[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml' 2022-03-08T23:00:09.105 INFO:teuthology.orchestra.run.smithi135.stdout:[preflight] Reading configuration from the cluster... 2022-03-08T23:00:09.105 INFO:teuthology.orchestra.run.smithi135.stdout:[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml' 2022-03-08T23:00:09.464 INFO:teuthology.orchestra.run.smithi106.stderr:W0308 23:00:09.476284 10688 utils.go:69] The recommended value for "resolvConf" in "KubeletConfiguration" is: /run/systemd/resolve/resolv.conf; the provided value is: /run/systemd/resolve/resolv.conf 2022-03-08T23:00:09.465 INFO:teuthology.orchestra.run.smithi135.stderr:W0308 23:00:09.466343 10667 utils.go:69] The recommended value for "resolvConf" in "KubeletConfiguration" is: /run/systemd/resolve/resolv.conf; the provided value is: /run/systemd/resolve/resolv.conf 2022-03-08T23:00:09.482 INFO:teuthology.orchestra.run.smithi106.stdout:[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" 2022-03-08T23:00:09.482 INFO:teuthology.orchestra.run.smithi106.stdout:[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" 2022-03-08T23:00:09.482 INFO:teuthology.orchestra.run.smithi106.stdout:[kubelet-start] Starting the kubelet 2022-03-08T23:00:09.484 INFO:teuthology.orchestra.run.smithi135.stdout:[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" 2022-03-08T23:00:09.484 INFO:teuthology.orchestra.run.smithi135.stdout:[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" 2022-03-08T23:00:09.484 INFO:teuthology.orchestra.run.smithi135.stdout:[kubelet-start] Starting the kubelet 2022-03-08T23:00:09.689 INFO:teuthology.orchestra.run.smithi106.stdout:[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap... 2022-03-08T23:00:09.716 INFO:teuthology.orchestra.run.smithi135.stdout:[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap... 2022-03-08T23:00:15.229 INFO:teuthology.orchestra.run.smithi106.stdout: 2022-03-08T23:00:15.229 INFO:teuthology.orchestra.run.smithi106.stdout:This node has joined the cluster: 2022-03-08T23:00:15.229 INFO:teuthology.orchestra.run.smithi106.stdout:* Certificate signing request was sent to apiserver and a response was received. 2022-03-08T23:00:15.230 INFO:teuthology.orchestra.run.smithi106.stdout:* The Kubelet was informed of the new secure connection details. 2022-03-08T23:00:15.230 INFO:teuthology.orchestra.run.smithi106.stdout: 2022-03-08T23:00:15.230 INFO:teuthology.orchestra.run.smithi106.stdout:Run 'kubectl get nodes' on the control-plane to see this node join the cluster. 2022-03-08T23:00:15.230 INFO:teuthology.orchestra.run.smithi106.stdout: 2022-03-08T23:00:23.736 INFO:teuthology.orchestra.run.smithi135.stdout: 2022-03-08T23:00:23.736 INFO:teuthology.orchestra.run.smithi135.stdout:This node has joined the cluster: 2022-03-08T23:00:23.737 INFO:teuthology.orchestra.run.smithi135.stdout:* Certificate signing request was sent to apiserver and a response was received. 2022-03-08T23:00:23.737 INFO:teuthology.orchestra.run.smithi135.stdout:* The Kubelet was informed of the new secure connection details. 2022-03-08T23:00:23.737 INFO:teuthology.orchestra.run.smithi135.stdout: 2022-03-08T23:00:23.737 INFO:teuthology.orchestra.run.smithi135.stdout:Run 'kubectl get nodes' on the control-plane to see this node join the cluster. 2022-03-08T23:00:23.738 INFO:teuthology.orchestra.run.smithi135.stdout: 2022-03-08T23:00:23.743 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2022-03-08T23:00:23.743 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/etc/kubernetes/admin.conf of=/dev/stdout 2022-03-08T23:00:23.754 INFO:tasks.kubeadm:Setting up kubectl 2022-03-08T23:00:23.755 DEBUG:teuthology.orchestra.run.smithi077:> mkdir -p .kube && sudo mkdir -p /root/.kube 2022-03-08T23:00:23.767 DEBUG:teuthology.orchestra.run.smithi106:> mkdir -p .kube && sudo mkdir -p /root/.kube 2022-03-08T23:00:23.784 DEBUG:teuthology.orchestra.run.smithi135:> mkdir -p .kube && sudo mkdir -p /root/.kube 2022-03-08T23:00:23.802 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2022-03-08T23:00:23.803 DEBUG:teuthology.orchestra.run.smithi077:> dd of=.kube/config 2022-03-08T23:00:23.810 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2022-03-08T23:00:23.810 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/root/.kube/config 2022-03-08T23:00:23.822 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2022-03-08T23:00:23.822 DEBUG:teuthology.orchestra.run.smithi106:> dd of=.kube/config 2022-03-08T23:00:23.830 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2022-03-08T23:00:23.830 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd of=/root/.kube/config 2022-03-08T23:00:23.845 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2022-03-08T23:00:23.846 DEBUG:teuthology.orchestra.run.smithi135:> dd of=.kube/config 2022-03-08T23:00:23.853 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2022-03-08T23:00:23.854 DEBUG:teuthology.orchestra.run.smithi135:> sudo dd of=/root/.kube/config 2022-03-08T23:00:23.912 DEBUG:teuthology.orchestra.run.smithi077:> curl https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml 2022-03-08T23:00:23.929 INFO:teuthology.orchestra.run.smithi077.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2022-03-08T23:00:23.929 INFO:teuthology.orchestra.run.smithi077.stderr: Dload Upload Total Spent Left Speed 2022-03-08T23:00:24.074 INFO:teuthology.orchestra.run.smithi077.stdout:--- 2022-03-08T23:00:24.079 INFO:teuthology.orchestra.run.smithi077.stdout:apiVersion: policy/v1beta1 2022-03-08T23:00:24.079 INFO:teuthology.orchestra.run.smithi077.stdout:kind: PodSecurityPolicy 2022-03-08T23:00:24.080 INFO:teuthology.orchestra.run.smithi077.stdout:metadata: 2022-03-08T23:00:24.080 INFO:teuthology.orchestra.run.smithi077.stdout: name: psp.flannel.unprivileged 2022-03-08T23:00:24.080 INFO:teuthology.orchestra.run.smithi077.stdout: annotations: 2022-03-08T23:00:24.080 INFO:teuthology.orchestra.run.smithi077.stdout: seccomp.security.alpha.kubernetes.io/allowedProfileNames: docker/default 2022-03-08T23:00:24.081 INFO:teuthology.orchestra.run.smithi077.stdout: seccomp.security.alpha.kubernetes.io/defaultProfileName: docker/default 2022-03-08T23:00:24.081 INFO:teuthology.orchestra.run.smithi077.stdout: apparmor.security.beta.kubernetes.io/allowedProfileNames: runtime/default 2022-03-08T23:00:24.081 INFO:teuthology.orchestra.run.smithi077.stdout: apparmor.security.beta.kubernetes.io/defaultProfileName: runtime/default 2022-03-08T23:00:24.081 INFO:teuthology.orchestra.run.smithi077.stdout:spec: 2022-03-08T23:00:24.082 INFO:teuthology.orchestra.run.smithi077.stdout: privileged: false 2022-03-08T23:00:24.082 INFO:teuthology.orchestra.run.smithi077.stdout: volumes: 2022-03-08T23:00:24.082 INFO:teuthology.orchestra.run.smithi077.stdout: - configMap 2022-03-08T23:00:24.082 INFO:teuthology.orchestra.run.smithi077.stdout: - secret 2022-03-08T23:00:24.082 INFO:teuthology.orchestra.run.smithi077.stdout: - emptyDir 2022-03-08T23:00:24.083 INFO:teuthology.orchestra.run.smithi077.stdout: - hostPath 2022-03-08T23:00:24.083 INFO:teuthology.orchestra.run.smithi077.stdout: allowedHostPaths: 2022-03-08T23:00:24.083 INFO:teuthology.orchestra.run.smithi077.stdout: - pathPrefix: "/etc/cni/net.d" 2022-03-08T23:00:24.083 INFO:teuthology.orchestra.run.smithi077.stdout: - pathPrefix: "/etc/kube-flannel" 2022-03-08T23:00:24.084 INFO:teuthology.orchestra.run.smithi077.stdout: - pathPrefix: "/run/flannel" 2022-03-08T23:00:24.084 INFO:teuthology.orchestra.run.smithi077.stdout: readOnlyRootFilesystem: false 2022-03-08T23:00:24.084 INFO:teuthology.orchestra.run.smithi077.stdout: # Users and groups 2022-03-08T23:00:24.084 INFO:teuthology.orchestra.run.smithi077.stdout: runAsUser: 2022-03-08T23:00:24.085 INFO:teuthology.orchestra.run.smithi077.stdout: rule: RunAsAny 2022-03-08T23:00:24.085 INFO:teuthology.orchestra.run.smithi077.stdout: supplementalGroups: 2022-03-08T23:00:24.085 INFO:teuthology.orchestra.run.smithi077.stdout: rule: RunAsAny 2022-03-08T23:00:24.085 INFO:teuthology.orchestra.run.smithi077.stdout: fsGroup: 2022-03-08T23:00:24.085 INFO:teuthology.orchestra.run.smithi077.stdout: rule: RunAsAny 2022-03-08T23:00:24.086 INFO:teuthology.orchestra.run.smithi077.stdout: # Privilege Escalation 2022-03-08T23:00:24.086 INFO:teuthology.orchestra.run.smithi077.stdout: allowPrivilegeEscalation: false 2022-03-08T23:00:24.086 INFO:teuthology.orchestra.run.smithi077.stdout: defaultAllowPrivilegeEscalation: false 2022-03-08T23:00:24.086 INFO:teuthology.orchestra.run.smithi077.stdout: # Capabilities 2022-03-08T23:00:24.087 INFO:teuthology.orchestra.run.smithi077.stdout: allowedCapabilities: ['NET_ADMIN', 'NET_RAW'] 2022-03-08T23:00:24.087 INFO:teuthology.orchestra.run.smithi077.stdout: defaultAddCapabilities: [] 2022-03-08T23:00:24.087 INFO:teuthology.orchestra.run.smithi077.stdout: requiredDropCapabilities: [] 2022-03-08T23:00:24.087 INFO:teuthology.orchestra.run.smithi077.stdout: # Host namespaces 2022-03-08T23:00:24.087 INFO:teuthology.orchestra.run.smithi077.stdout: hostPID: false 2022-03-08T23:00:24.088 INFO:teuthology.orchestra.run.smithi077.stdout: hostIPC: false 2022-03-08T23:00:24.088 INFO:teuthology.orchestra.run.smithi077.stdout: hostNetwork: true 2022-03-08T23:00:24.088 INFO:teuthology.orchestra.run.smithi077.stdout: hostPorts: 2022-03-08T23:00:24.088 INFO:teuthology.orchestra.run.smithi077.stdout: - min: 0 2022-03-08T23:00:24.089 INFO:teuthology.orchestra.run.smithi077.stdout: max: 65535 2022-03-08T23:00:24.089 INFO:teuthology.orchestra.run.smithi077.stdout: # SELinux 2022-03-08T23:00:24.089 INFO:teuthology.orchestra.run.smithi077.stdout: seLinux: 2022-03-08T23:00:24.089 INFO:teuthology.orchestra.run.smithi077.stdout: # SELinux is unused in CaaSP 2022-03-08T23:00:24.089 INFO:teuthology.orchestra.run.smithi077.stdout: rule: 'RunAsAny' 2022-03-08T23:00:24.090 INFO:teuthology.orchestra.run.smithi077.stdout:--- 2022-03-08T23:00:24.090 INFO:teuthology.orchestra.run.smithi077.stdout:kind: ClusterRole 2022-03-08T23:00:24.090 INFO:teuthology.orchestra.run.smithi077.stdout:apiVersion: rbac.authorization.k8s.io/v1 2022-03-08T23:00:24.090 INFO:teuthology.orchestra.run.smithi077.stdout:metadata: 2022-03-08T23:00:24.091 INFO:teuthology.orchestra.run.smithi077.stdout: name: flannel 2022-03-08T23:00:24.091 INFO:teuthology.orchestra.run.smithi077.stdout:rules: 2022-03-08T23:00:24.091 INFO:teuthology.orchestra.run.smithi077.stdout:- apiGroups: ['extensions'] 2022-03-08T23:00:24.091 INFO:teuthology.orchestra.run.smithi077.stdout: resources: ['podsecuritypolicies'] 2022-03-08T23:00:24.092 INFO:teuthology.orchestra.run.smithi077.stdout: verbs: ['use'] 2022-03-08T23:00:24.092 INFO:teuthology.orchestra.run.smithi077.stdout: resourceNames: ['psp.flannel.unprivileged'] 2022-03-08T23:00:24.092 INFO:teuthology.orchestra.run.smithi077.stdout:- apiGroups: 2022-03-08T23:00:24.092 INFO:teuthology.orchestra.run.smithi077.stdout: - "" 2022-03-08T23:00:24.092 INFO:teuthology.orchestra.run.smithi077.stdout: resources: 2022-03-08T23:00:24.093 INFO:teuthology.orchestra.run.smithi077.stdout: - pods 2022-03-08T23:00:24.093 INFO:teuthology.orchestra.run.smithi077.stdout: verbs: 2022-03-08T23:00:24.093 INFO:teuthology.orchestra.run.smithi077.stdout: - get 2022-03-08T23:00:24.093 INFO:teuthology.orchestra.run.smithi077.stdout:- apiGroups: 2022-03-08T23:00:24.094 INFO:teuthology.orchestra.run.smithi077.stdout: - "" 2022-03-08T23:00:24.094 INFO:teuthology.orchestra.run.smithi077.stdout: resources: 2022-03-08T23:00:24.094 INFO:teuthology.orchestra.run.smithi077.stdout: - nodes 2022-03-08T23:00:24.094 INFO:teuthology.orchestra.run.smithi077.stdout: verbs: 2022-03-08T23:00:24.094 INFO:teuthology.orchestra.run.smithi077.stdout: - list 2022-03-08T23:00:24.095 INFO:teuthology.orchestra.run.smithi077.stdout: - watch 2022-03-08T23:00:24.095 INFO:teuthology.orchestra.run.smithi077.stdout:- apiGroups: 2022-03-08T23:00:24.095 INFO:teuthology.orchestra.run.smithi077.stdout: - "" 2022-03-08T23:00:24.095 INFO:teuthology.orchestra.run.smithi077.stdout: resources: 2022-03-08T23:00:24.095 INFO:teuthology.orchestra.run.smithi077.stdout: - nodes/status 2022-03-08T23:00:24.096 INFO:teuthology.orchestra.run.smithi077.stdout: verbs: 2022-03-08T23:00:24.096 INFO:teuthology.orchestra.run.smithi077.stdout: - patch 2022-03-08T23:00:24.096 INFO:teuthology.orchestra.run.smithi077.stdout:--- 2022-03-08T23:00:24.096 INFO:teuthology.orchestra.run.smithi077.stdout:kind: ClusterRoleBinding 2022-03-08T23:00:24.097 INFO:teuthology.orchestra.run.smithi077.stdout:apiVersion: rbac.authorization.k8s.io/v1 2022-03-08T23:00:24.097 INFO:teuthology.orchestra.run.smithi077.stdout:metadata: 2022-03-08T23:00:24.097 INFO:teuthology.orchestra.run.smithi077.stdout: name: flannel 2022-03-08T23:00:24.097 INFO:teuthology.orchestra.run.smithi077.stdout:roleRef: 2022-03-08T23:00:24.098 INFO:teuthology.orchestra.run.smithi077.stdout: apiGroup: rbac.authorization.k8s.io 2022-03-08T23:00:24.098 INFO:teuthology.orchestra.run.smithi077.stdout: kind: ClusterRole 2022-03-08T23:00:24.098 INFO:teuthology.orchestra.run.smithi077.stdout: name: flannel 2022-03-08T23:00:24.098 INFO:teuthology.orchestra.run.smithi077.stdout:subjects: 2022-03-08T23:00:24.098 INFO:teuthology.orchestra.run.smithi077.stdout:- kind: ServiceAccount 2022-03-08T23:00:24.099 INFO:teuthology.orchestra.run.smithi077.stdout: name: flannel 2022-03-08T23:00:24.099 INFO:teuthology.orchestra.run.smithi077.stdout: namespace: kube-system 2022-03-08T23:00:24.099 INFO:teuthology.orchestra.run.smithi077.stdout:--- 2022-03-08T23:00:24.099 INFO:teuthology.orchestra.run.smithi077.stdout:apiVersion: v1 2022-03-08T23:00:24.100 INFO:teuthology.orchestra.run.smithi077.stdout:kind: ServiceAccount 2022-03-08T23:00:24.100 INFO:teuthology.orchestra.run.smithi077.stdout:metadata: 2022-03-08T23:00:24.100 INFO:teuthology.orchestra.run.smithi077.stdout: name: flannel 2022-03-08T23:00:24.100 INFO:teuthology.orchestra.run.smithi077.stdout: namespace: kube-system 2022-03-08T23:00:24.100 INFO:teuthology.orchestra.run.smithi077.stdout:--- 2022-03-08T23:00:24.101 INFO:teuthology.orchestra.run.smithi077.stdout:kind: ConfigMap 2022-03-08T23:00:24.101 INFO:teuthology.orchestra.run.smithi077.stdout:apiVersion: v1 2022-03-08T23:00:24.101 INFO:teuthology.orchestra.run.smithi077.stdout:metadata: 2022-03-08T23:00:24.101 INFO:teuthology.orchestra.run.smithi077.stdout: name: kube-flannel-cfg 2022-03-08T23:00:24.101 INFO:teuthology.orchestra.run.smithi077.stdout: namespace: kube-system 2022-03-08T23:00:24.102 INFO:teuthology.orchestra.run.smithi077.stdout: labels: 2022-03-08T23:00:24.102 INFO:teuthology.orchestra.run.smithi077.stdout: tier: node 2022-03-08T23:00:24.102 INFO:teuthology.orchestra.run.smithi077.stdout: app: flannel 2022-03-08T23:00:24.102 INFO:teuthology.orchestra.run.smithi077.stdout:data: 2022-03-08T23:00:24.103 INFO:teuthology.orchestra.run.smithi077.stdout: cni-conf.json: | 2022-03-08T23:00:24.103 INFO:teuthology.orchestra.run.smithi077.stdout: { 2022-03-08T23:00:24.103 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "cbr0", 2022-03-08T23:00:24.103 INFO:teuthology.orchestra.run.smithi077.stdout: "cniVersion": "0.3.1", 2022-03-08T23:00:24.103 INFO:teuthology.orchestra.run.smithi077.stdout: "plugins": [ 2022-03-08T23:00:24.104 INFO:teuthology.orchestra.run.smithi077.stdout: { 2022-03-08T23:00:24.104 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "flannel", 2022-03-08T23:00:24.104 INFO:teuthology.orchestra.run.smithi077.stdout: "delegate": { 2022-03-08T23:00:24.104 INFO:teuthology.orchestra.run.smithi077.stdout: "hairpinMode": true, 2022-03-08T23:00:24.105 INFO:teuthology.orchestra.run.smithi077.stdout: "isDefaultGateway": true 2022-03-08T23:00:24.105 INFO:teuthology.orchestra.run.smithi077.stdout: } 2022-03-08T23:00:24.105 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2022-03-08T23:00:24.105 INFO:teuthology.orchestra.run.smithi077.stdout: { 2022-03-08T23:00:24.105 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "portmap", 2022-03-08T23:00:24.106 INFO:teuthology.orchestra.run.smithi077.stdout: "capabilities": { 2022-03-08T23:00:24.106 INFO:teuthology.orchestra.run.smithi077.stdout: "portMappings": true 2022-03-08T23:00:24.106 INFO:teuthology.orchestra.run.smithi077.stdout: } 2022-03-08T23:00:24.106 INFO:teuthology.orchestra.run.smithi077.stdout: } 2022-03-08T23:00:24.106 INFO:teuthology.orchestra.run.smithi077.stdout: ] 2022-03-08T23:00:24.107 INFO:teuthology.orchestra.run.smithi077.stdout: } 2022-03-08T23:00:24.107 INFO:teuthology.orchestra.run.smithi077.stdout: net-conf.json: | 2022-03-08T23:00:24.107 INFO:teuthology.orchestra.run.smithi077.stdout: { 2022-03-08T23:00:24.107 INFO:teuthology.orchestra.run.smithi077.stdout: "Network": "10.244.0.0/16", 2022-03-08T23:00:24.108 INFO:teuthology.orchestra.run.smithi077.stdout: "Backend": { 2022-03-08T23:00:24.108 INFO:teuthology.orchestra.run.smithi077.stdout: "Type": "vxlan" 2022-03-08T23:00:24.108 INFO:teuthology.orchestra.run.smithi077.stdout: } 2022-03-08T23:00:24.108 INFO:teuthology.orchestra.run.smithi077.stdout: } 2022-03-08T23:00:24.108 INFO:teuthology.orchestra.run.smithi077.stdout:--- 2022-03-08T23:00:24.109 INFO:teuthology.orchestra.run.smithi077.stdout:apiVersion: apps/v1 2022-03-08T23:00:24.109 INFO:teuthology.orchestra.run.smithi077.stdout:kind: DaemonSet 2022-03-08T23:00:24.109 INFO:teuthology.orchestra.run.smithi077.stdout:metadata: 2022-03-08T23:00:24.109 INFO:teuthology.orchestra.run.smithi077.stdout: name: kube-flannel-ds 2022-03-08T23:00:24.109 INFO:teuthology.orchestra.run.smithi077.stdout: namespace: kube-system 2022-03-08T23:00:24.110 INFO:teuthology.orchestra.run.smithi077.stdout: labels: 2022-03-08T23:00:24.110 INFO:teuthology.orchestra.run.smithi077.stdout: tier: node 2022-03-08T23:00:24.110 INFO:teuthology.orchestra.run.smithi077.stdout: app: flannel 2022-03-08T23:00:24.110 INFO:teuthology.orchestra.run.smithi077.stdout:spec: 2022-03-08T23:00:24.111 INFO:teuthology.orchestra.run.smithi077.stdout: selector: 2022-03-08T23:00:24.111 INFO:teuthology.orchestra.run.smithi077.stdout: matchLabels: 2022-03-08T23:00:24.111 INFO:teuthology.orchestra.run.smithi077.stdout: app: flannel 2022-03-08T23:00:24.111 INFO:teuthology.orchestra.run.smithi077.stdout: template: 2022-03-08T23:00:24.112 INFO:teuthology.orchestra.run.smithi077.stdout: metadata: 2022-03-08T23:00:24.112 INFO:teuthology.orchestra.run.smithi077.stdout: labels: 2022-03-08T23:00:24.112 INFO:teuthology.orchestra.run.smithi077.stdout: tier: node 2022-03-08T23:00:24.112 INFO:teuthology.orchestra.run.smithi077.stdout: app: flannel 2022-03-08T23:00:24.113 INFO:teuthology.orchestra.run.smithi077.stdout: spec: 2022-03-08T23:00:24.113 INFO:teuthology.orchestra.run.smithi077.stdout: affinity: 2022-03-08T23:00:24.113 INFO:teuthology.orchestra.run.smithi077.stdout: nodeAffinity: 2022-03-08T23:00:24.113 INFO:teuthology.orchestra.run.smithi077.stdout: requiredDuringSchedulingIgnoredDuringExecution: 2022-03-08T23:00:24.113 INFO:teuthology.orchestra.run.smithi077.stdout: nodeSelectorTerms: 2022-03-08T23:00:24.114 INFO:teuthology.orchestra.run.smithi077.stdout: - matchExpressions: 2022-03-08T23:00:24.114 INFO:teuthology.orchestra.run.smithi077.stdout: - key: kubernetes.io/os 2022-03-08T23:00:24.114 INFO:teuthology.orchestra.run.smithi077.stdout: operator: In 2022-03-08T23:00:24.114 INFO:teuthology.orchestra.run.smithi077.stdout: values: 2022-03-08T23:00:24.115 INFO:teuthology.orchestra.run.smithi077.stdout: - linux 2022-03-08T23:00:24.115 INFO:teuthology.orchestra.run.smithi077.stdout: hostNetwork: true 2022-03-08T23:00:24.115 INFO:teuthology.orchestra.run.smithi077.stdout: priorityClassName: system-node-critical 2022-03-08T23:00:24.115 INFO:teuthology.orchestra.run.smithi077.stdout: tolerations: 2022-03-08T23:00:24.115 INFO:teuthology.orchestra.run.smithi077.stdout: - operator: Exists 2022-03-08T23:00:24.116 INFO:teuthology.orchestra.run.smithi077.stdout: effect: NoSchedule 2022-03-08T23:00:24.116 INFO:teuthology.orchestra.run.smithi077.stdout: serviceAccountName: flannel 2022-03-08T23:00:24.116 INFO:teuthology.orchestra.run.smithi077.stdout: initContainers: 2022-03-08T23:00:24.116 INFO:teuthology.orchestra.run.smithi077.stdout: - name: install-cni-plugin 2022-03-08T23:00:24.117 INFO:teuthology.orchestra.run.smithi077.stdout: #image: flannelcni/flannel-cni-plugin:v1.0.1 for ppc64le and mips64le (dockerhub limitations may apply) 2022-03-08T23:00:24.117 INFO:teuthology.orchestra.run.smithi077.stdout: image: rancher/mirrored-flannelcni-flannel-cni-plugin:v1.0.1 2022-03-08T23:00:24.117 INFO:teuthology.orchestra.run.smithi077.stdout: command: 2022-03-08T23:00:24.117 INFO:teuthology.orchestra.run.smithi077.stdout: - cp 2022-03-08T23:00:24.117 INFO:teuthology.orchestra.run.smithi077.stdout: args: 2022-03-08T23:00:24.118 INFO:teuthology.orchestra.run.smithi077.stdout: - -f 2022-03-08T23:00:24.118 INFO:teuthology.orchestra.run.smithi077.stdout: - /flannel 2022-03-08T23:00:24.118 INFO:teuthology.orchestra.run.smithi077.stdout: - /opt/cni/bin/flannel 2022-03-08T23:00:24.118 INFO:teuthology.orchestra.run.smithi077.stdout: volumeMounts: 2022-03-08T23:00:24.118 INFO:teuthology.orchestra.run.smithi077.stdout: - name: cni-plugin 2022-03-08T23:00:24.119 INFO:teuthology.orchestra.run.smithi077.stdout: mountPath: /opt/cni/bin 2022-03-08T23:00:24.119 INFO:teuthology.orchestra.run.smithi077.stdout: - name: install-cni 2022-03-08T23:00:24.119 INFO:teuthology.orchestra.run.smithi077.stdout: #image: flannelcni/flannel:v0.17.0 for ppc64le and mips64le (dockerhub limitations may apply) 2022-03-08T23:00:24.119 INFO:teuthology.orchestra.run.smithi077.stdout: image: rancher/mirrored-flannelcni-flannel:v0.17.0 2022-03-08T23:00:24.120 INFO:teuthology.orchestra.run.smithi077.stdout: command: 2022-03-08T23:00:24.120 INFO:teuthology.orchestra.run.smithi077.stdout: - cp 2022-03-08T23:00:24.120 INFO:teuthology.orchestra.run.smithi077.stdout: args: 2022-03-08T23:00:24.120 INFO:teuthology.orchestra.run.smithi077.stdout: - -f 2022-03-08T23:00:24.121 INFO:teuthology.orchestra.run.smithi077.stdout: - /etc/kube-flannel/cni-conf.json 2022-03-08T23:00:24.121 INFO:teuthology.orchestra.run.smithi077.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 5692 100 5692 0 0 39255 0 --:--:-- --:--:-- --:--:-- 39255 2022-03-08T23:00:24.122 INFO:teuthology.orchestra.run.smithi077.stdout: - /etc/cni/net.d/10-flannel.conflist 2022-03-08T23:00:24.123 INFO:teuthology.orchestra.run.smithi077.stdout: volumeMounts: 2022-03-08T23:00:24.123 INFO:teuthology.orchestra.run.smithi077.stdout: - name: cni 2022-03-08T23:00:24.123 INFO:teuthology.orchestra.run.smithi077.stdout: mountPath: /etc/cni/net.d 2022-03-08T23:00:24.123 INFO:teuthology.orchestra.run.smithi077.stdout: - name: flannel-cfg 2022-03-08T23:00:24.124 INFO:teuthology.orchestra.run.smithi077.stdout: mountPath: /etc/kube-flannel/ 2022-03-08T23:00:24.124 INFO:teuthology.orchestra.run.smithi077.stdout: containers: 2022-03-08T23:00:24.124 INFO:teuthology.orchestra.run.smithi077.stdout: - name: kube-flannel 2022-03-08T23:00:24.124 INFO:teuthology.orchestra.run.smithi077.stdout: #image: flannelcni/flannel:v0.17.0 for ppc64le and mips64le (dockerhub limitations may apply) 2022-03-08T23:00:24.124 INFO:teuthology.orchestra.run.smithi077.stdout: image: rancher/mirrored-flannelcni-flannel:v0.17.0 2022-03-08T23:00:24.125 INFO:teuthology.orchestra.run.smithi077.stdout: command: 2022-03-08T23:00:24.125 INFO:teuthology.orchestra.run.smithi077.stdout: - /opt/bin/flanneld 2022-03-08T23:00:24.125 INFO:teuthology.orchestra.run.smithi077.stdout: args: 2022-03-08T23:00:24.125 INFO:teuthology.orchestra.run.smithi077.stdout: - --ip-masq 2022-03-08T23:00:24.126 INFO:teuthology.orchestra.run.smithi077.stdout: - --kube-subnet-mgr 2022-03-08T23:00:24.126 INFO:teuthology.orchestra.run.smithi077.stdout: resources: 2022-03-08T23:00:24.126 INFO:teuthology.orchestra.run.smithi077.stdout: requests: 2022-03-08T23:00:24.126 INFO:teuthology.orchestra.run.smithi077.stdout: cpu: "100m" 2022-03-08T23:00:24.127 INFO:teuthology.orchestra.run.smithi077.stdout: memory: "50Mi" 2022-03-08T23:00:24.127 INFO:teuthology.orchestra.run.smithi077.stdout: limits: 2022-03-08T23:00:24.127 INFO:teuthology.orchestra.run.smithi077.stdout: cpu: "100m" 2022-03-08T23:00:24.127 INFO:teuthology.orchestra.run.smithi077.stdout: memory: "50Mi" 2022-03-08T23:00:24.127 INFO:teuthology.orchestra.run.smithi077.stdout: securityContext: 2022-03-08T23:00:24.128 INFO:teuthology.orchestra.run.smithi077.stdout: privileged: false 2022-03-08T23:00:24.128 INFO:teuthology.orchestra.run.smithi077.stdout: capabilities: 2022-03-08T23:00:24.128 INFO:teuthology.orchestra.run.smithi077.stdout: add: ["NET_ADMIN", "NET_RAW"] 2022-03-08T23:00:24.128 INFO:teuthology.orchestra.run.smithi077.stdout: env: 2022-03-08T23:00:24.129 INFO:teuthology.orchestra.run.smithi077.stdout: - name: POD_NAME 2022-03-08T23:00:24.129 INFO:teuthology.orchestra.run.smithi077.stdout: valueFrom: 2022-03-08T23:00:24.129 INFO:teuthology.orchestra.run.smithi077.stdout: fieldRef: 2022-03-08T23:00:24.129 INFO:teuthology.orchestra.run.smithi077.stdout: fieldPath: metadata.name 2022-03-08T23:00:24.129 INFO:teuthology.orchestra.run.smithi077.stdout: - name: POD_NAMESPACE 2022-03-08T23:00:24.130 INFO:teuthology.orchestra.run.smithi077.stdout: valueFrom: 2022-03-08T23:00:24.130 INFO:teuthology.orchestra.run.smithi077.stdout: fieldRef: 2022-03-08T23:00:24.130 INFO:teuthology.orchestra.run.smithi077.stdout: fieldPath: metadata.namespace 2022-03-08T23:00:24.130 INFO:teuthology.orchestra.run.smithi077.stdout: volumeMounts: 2022-03-08T23:00:24.131 INFO:teuthology.orchestra.run.smithi077.stdout: - name: run 2022-03-08T23:00:24.131 INFO:teuthology.orchestra.run.smithi077.stdout: mountPath: /run/flannel 2022-03-08T23:00:24.131 INFO:teuthology.orchestra.run.smithi077.stdout: - name: flannel-cfg 2022-03-08T23:00:24.131 INFO:teuthology.orchestra.run.smithi077.stdout: mountPath: /etc/kube-flannel/ 2022-03-08T23:00:24.131 INFO:teuthology.orchestra.run.smithi077.stdout: - name: xtables-lock 2022-03-08T23:00:24.132 INFO:teuthology.orchestra.run.smithi077.stdout: mountPath: /run/xtables.lock 2022-03-08T23:00:24.132 INFO:teuthology.orchestra.run.smithi077.stdout: volumes: 2022-03-08T23:00:24.132 INFO:teuthology.orchestra.run.smithi077.stdout: - name: run 2022-03-08T23:00:24.132 INFO:teuthology.orchestra.run.smithi077.stdout: hostPath: 2022-03-08T23:00:24.133 INFO:teuthology.orchestra.run.smithi077.stdout: path: /run/flannel 2022-03-08T23:00:24.133 INFO:teuthology.orchestra.run.smithi077.stdout: - name: cni-plugin 2022-03-08T23:00:24.133 INFO:teuthology.orchestra.run.smithi077.stdout: hostPath: 2022-03-08T23:00:24.133 INFO:teuthology.orchestra.run.smithi077.stdout: path: /opt/cni/bin 2022-03-08T23:00:24.133 INFO:teuthology.orchestra.run.smithi077.stdout: - name: cni 2022-03-08T23:00:24.134 INFO:teuthology.orchestra.run.smithi077.stdout: hostPath: 2022-03-08T23:00:24.134 INFO:teuthology.orchestra.run.smithi077.stdout: path: /etc/cni/net.d 2022-03-08T23:00:24.134 INFO:teuthology.orchestra.run.smithi077.stdout: - name: flannel-cfg 2022-03-08T23:00:24.134 INFO:teuthology.orchestra.run.smithi077.stdout: configMap: 2022-03-08T23:00:24.134 INFO:teuthology.orchestra.run.smithi077.stdout: name: kube-flannel-cfg 2022-03-08T23:00:24.135 INFO:teuthology.orchestra.run.smithi077.stdout: - name: xtables-lock 2022-03-08T23:00:24.135 INFO:teuthology.orchestra.run.smithi077.stdout: hostPath: 2022-03-08T23:00:24.135 INFO:teuthology.orchestra.run.smithi077.stdout: path: /run/xtables.lock 2022-03-08T23:00:24.135 INFO:teuthology.orchestra.run.smithi077.stdout: type: FileOrCreate 2022-03-08T23:00:24.163 INFO:tasks.kubeadm:Updating {'kind': 'ConfigMap', 'apiVersion': 'v1', 'metadata': {'name': 'kube-flannel-cfg', 'namespace': 'kube-system', 'labels': {'tier': 'node', 'app': 'flannel'}}, 'data': {'cni-conf.json': '{\n "name": "cbr0",\n "cniVersion": "0.3.1",\n "plugins": [\n {\n "type": "flannel",\n "delegate": {\n "hairpinMode": true,\n "isDefaultGateway": true\n }\n },\n {\n "type": "portmap",\n "capabilities": {\n "portMappings": true\n }\n }\n ]\n}\n', 'net-conf.json': '{\n "Network": "10.244.0.0/16",\n "Backend": {\n "Type": "vxlan"\n }\n}\n'}} 2022-03-08T23:00:24.163 INFO:tasks.kubeadm:Now {'kind': 'ConfigMap', 'apiVersion': 'v1', 'metadata': {'name': 'kube-flannel-cfg', 'namespace': 'kube-system', 'labels': {'tier': 'node', 'app': 'flannel'}}, 'data': {'cni-conf.json': '{\n "name": "cbr0",\n "cniVersion": "0.3.1",\n "plugins": [\n {\n "type": "flannel",\n "delegate": {\n "hairpinMode": true,\n "isDefaultGateway": true\n }\n },\n {\n "type": "portmap",\n "capabilities": {\n "portMappings": true\n }\n }\n ]\n}\n', 'net-conf.json': '{\n "Network": "10.250.96.0/21",\n "Backend": {\n "Type": "vxlan"\n }\n}\n'}} 2022-03-08T23:00:24.178 DEBUG:tasks.kubeadm:Flannel: apiVersion: policy/v1beta1 kind: PodSecurityPolicy metadata: annotations: apparmor.security.beta.kubernetes.io/allowedProfileNames: runtime/default apparmor.security.beta.kubernetes.io/defaultProfileName: runtime/default seccomp.security.alpha.kubernetes.io/allowedProfileNames: docker/default seccomp.security.alpha.kubernetes.io/defaultProfileName: docker/default name: psp.flannel.unprivileged spec: allowPrivilegeEscalation: false allowedCapabilities: - NET_ADMIN - NET_RAW allowedHostPaths: - pathPrefix: /etc/cni/net.d - pathPrefix: /etc/kube-flannel - pathPrefix: /run/flannel defaultAddCapabilities: [] defaultAllowPrivilegeEscalation: false fsGroup: rule: RunAsAny hostIPC: false hostNetwork: true hostPID: false hostPorts: - max: 65535 min: 0 privileged: false readOnlyRootFilesystem: false requiredDropCapabilities: [] runAsUser: rule: RunAsAny seLinux: rule: RunAsAny supplementalGroups: rule: RunAsAny volumes: - configMap - secret - emptyDir - hostPath --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: flannel rules: - apiGroups: - extensions resourceNames: - psp.flannel.unprivileged resources: - podsecuritypolicies verbs: - use - apiGroups: - '' resources: - pods verbs: - get - apiGroups: - '' resources: - nodes verbs: - list - watch - apiGroups: - '' resources: - nodes/status verbs: - patch --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: flannel roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: flannel subjects: - kind: ServiceAccount name: flannel namespace: kube-system --- apiVersion: v1 kind: ServiceAccount metadata: name: flannel namespace: kube-system --- apiVersion: v1 data: cni-conf.json: "{\n \"name\": \"cbr0\",\n \"cniVersion\": \"0.3.1\",\n \"plugins\"\ : [\n {\n \"type\": \"flannel\",\n \"delegate\": {\n \"hairpinMode\"\ : true,\n \"isDefaultGateway\": true\n }\n },\n {\n \"\ type\": \"portmap\",\n \"capabilities\": {\n \"portMappings\": true\n\ \ }\n }\n ]\n}\n" net-conf.json: "{\n \"Network\": \"10.250.96.0/21\",\n \"Backend\": {\n \"\ Type\": \"vxlan\"\n }\n}\n" kind: ConfigMap metadata: labels: app: flannel tier: node name: kube-flannel-cfg namespace: kube-system --- apiVersion: apps/v1 kind: DaemonSet metadata: labels: app: flannel tier: node name: kube-flannel-ds namespace: kube-system spec: selector: matchLabels: app: flannel template: metadata: labels: app: flannel tier: node spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/os operator: In values: - linux containers: - args: - --ip-masq - --kube-subnet-mgr command: - /opt/bin/flanneld env: - name: POD_NAME valueFrom: fieldRef: fieldPath: metadata.name - name: POD_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace image: rancher/mirrored-flannelcni-flannel:v0.17.0 name: kube-flannel resources: limits: cpu: 100m memory: 50Mi requests: cpu: 100m memory: 50Mi securityContext: capabilities: add: - NET_ADMIN - NET_RAW privileged: false volumeMounts: - mountPath: /run/flannel name: run - mountPath: /etc/kube-flannel/ name: flannel-cfg - mountPath: /run/xtables.lock name: xtables-lock hostNetwork: true initContainers: - args: - -f - /flannel - /opt/cni/bin/flannel command: - cp image: rancher/mirrored-flannelcni-flannel-cni-plugin:v1.0.1 name: install-cni-plugin volumeMounts: - mountPath: /opt/cni/bin name: cni-plugin - args: - -f - /etc/kube-flannel/cni-conf.json - /etc/cni/net.d/10-flannel.conflist command: - cp image: rancher/mirrored-flannelcni-flannel:v0.17.0 name: install-cni volumeMounts: - mountPath: /etc/cni/net.d name: cni - mountPath: /etc/kube-flannel/ name: flannel-cfg priorityClassName: system-node-critical serviceAccountName: flannel tolerations: - effect: NoSchedule operator: Exists volumes: - hostPath: path: /run/flannel name: run - hostPath: path: /opt/cni/bin name: cni-plugin - hostPath: path: /etc/cni/net.d name: cni - configMap: name: kube-flannel-cfg name: flannel-cfg - hostPath: path: /run/xtables.lock type: FileOrCreate name: xtables-lock 2022-03-08T23:00:24.178 DEBUG:teuthology.orchestra.run.smithi077:> kubectl apply -f - 2022-03-08T23:00:26.588 INFO:teuthology.orchestra.run.smithi077.stderr:Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+ 2022-03-08T23:00:26.594 INFO:teuthology.orchestra.run.smithi077.stdout:podsecuritypolicy.policy/psp.flannel.unprivileged created 2022-03-08T23:00:26.620 INFO:teuthology.orchestra.run.smithi077.stdout:clusterrole.rbac.authorization.k8s.io/flannel created 2022-03-08T23:00:26.629 INFO:teuthology.orchestra.run.smithi077.stdout:clusterrolebinding.rbac.authorization.k8s.io/flannel created 2022-03-08T23:00:26.639 INFO:teuthology.orchestra.run.smithi077.stdout:serviceaccount/flannel created 2022-03-08T23:00:26.647 INFO:teuthology.orchestra.run.smithi077.stdout:configmap/kube-flannel-cfg created 2022-03-08T23:00:26.706 INFO:teuthology.orchestra.run.smithi077.stdout:daemonset.apps/kube-flannel-ds created 2022-03-08T23:00:26.711 INFO:tasks.kubeadm:Scanning for scratch devices 2022-03-08T23:00:26.712 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2022-03-08T23:00:26.712 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/scratch_devs of=/dev/stdout 2022-03-08T23:00:26.719 INFO:tasks.kubeadm:Scratch devices on smithi077: ['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-03-08T23:00:26.719 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 2022-03-08T23:00:26.784 INFO:teuthology.orchestra.run.smithi077.stderr:10+0 records in 2022-03-08T23:00:26.784 INFO:teuthology.orchestra.run.smithi077.stderr:10+0 records out 2022-03-08T23:00:26.784 INFO:teuthology.orchestra.run.smithi077.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.01328 s, 790 MB/s 2022-03-08T23:00:26.786 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 2022-03-08T23:00:26.809 INFO:teuthology.orchestra.run.smithi077.stderr:10+0 records in 2022-03-08T23:00:26.810 INFO:teuthology.orchestra.run.smithi077.stderr:10+0 records out 2022-03-08T23:00:26.810 INFO:teuthology.orchestra.run.smithi077.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0131419 s, 798 MB/s 2022-03-08T23:00:26.811 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 2022-03-08T23:00:26.833 INFO:teuthology.orchestra.run.smithi077.stderr:10+0 records in 2022-03-08T23:00:26.833 INFO:teuthology.orchestra.run.smithi077.stderr:10+0 records out 2022-03-08T23:00:26.834 INFO:teuthology.orchestra.run.smithi077.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0123847 s, 847 MB/s 2022-03-08T23:00:26.835 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 2022-03-08T23:00:26.864 INFO:teuthology.orchestra.run.smithi077.stderr:10+0 records in 2022-03-08T23:00:26.865 INFO:teuthology.orchestra.run.smithi077.stderr:10+0 records out 2022-03-08T23:00:26.865 INFO:teuthology.orchestra.run.smithi077.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0182082 s, 576 MB/s 2022-03-08T23:00:26.866 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2022-03-08T23:00:26.867 DEBUG:teuthology.orchestra.run.smithi106:> dd if=/scratch_devs of=/dev/stdout 2022-03-08T23:00:26.875 INFO:tasks.kubeadm:Scratch devices on smithi106: ['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-03-08T23:00:26.876 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 2022-03-08T23:00:26.902 INFO:teuthology.orchestra.run.smithi106.stderr:10+0 records in 2022-03-08T23:00:26.902 INFO:teuthology.orchestra.run.smithi106.stderr:10+0 records out 2022-03-08T23:00:26.902 INFO:teuthology.orchestra.run.smithi106.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0147505 s, 711 MB/s 2022-03-08T23:00:26.904 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 2022-03-08T23:00:26.931 INFO:teuthology.orchestra.run.smithi106.stderr:10+0 records in 2022-03-08T23:00:26.931 INFO:teuthology.orchestra.run.smithi106.stderr:10+0 records out 2022-03-08T23:00:26.931 INFO:teuthology.orchestra.run.smithi106.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0147801 s, 709 MB/s 2022-03-08T23:00:26.933 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 2022-03-08T23:00:26.959 INFO:teuthology.orchestra.run.smithi106.stderr:10+0 records in 2022-03-08T23:00:26.959 INFO:teuthology.orchestra.run.smithi106.stderr:10+0 records out 2022-03-08T23:00:26.960 INFO:teuthology.orchestra.run.smithi106.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0149044 s, 704 MB/s 2022-03-08T23:00:26.961 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 2022-03-08T23:00:27.034 INFO:teuthology.orchestra.run.smithi106.stderr:10+0 records in 2022-03-08T23:00:27.035 INFO:teuthology.orchestra.run.smithi106.stderr:10+0 records out 2022-03-08T23:00:27.035 INFO:teuthology.orchestra.run.smithi106.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0179064 s, 586 MB/s 2022-03-08T23:00:27.036 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2022-03-08T23:00:27.037 DEBUG:teuthology.orchestra.run.smithi135:> dd if=/scratch_devs of=/dev/stdout 2022-03-08T23:00:27.045 INFO:tasks.kubeadm:Scratch devices on smithi135: ['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-03-08T23:00:27.045 DEBUG:teuthology.orchestra.run.smithi135:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 2022-03-08T23:00:27.077 INFO:teuthology.orchestra.run.smithi135.stderr:10+0 records in 2022-03-08T23:00:27.077 INFO:teuthology.orchestra.run.smithi135.stderr:10+0 records out 2022-03-08T23:00:27.078 INFO:teuthology.orchestra.run.smithi135.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0174337 s, 601 MB/s 2022-03-08T23:00:27.079 DEBUG:teuthology.orchestra.run.smithi135:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 2022-03-08T23:00:27.149 INFO:teuthology.orchestra.run.smithi135.stderr:10+0 records in 2022-03-08T23:00:27.150 INFO:teuthology.orchestra.run.smithi135.stderr:10+0 records out 2022-03-08T23:00:27.150 INFO:teuthology.orchestra.run.smithi135.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0160375 s, 654 MB/s 2022-03-08T23:00:27.151 DEBUG:teuthology.orchestra.run.smithi135:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 2022-03-08T23:00:27.178 INFO:teuthology.orchestra.run.smithi135.stderr:10+0 records in 2022-03-08T23:00:27.179 INFO:teuthology.orchestra.run.smithi135.stderr:10+0 records out 2022-03-08T23:00:27.179 INFO:teuthology.orchestra.run.smithi135.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0156443 s, 670 MB/s 2022-03-08T23:00:27.180 DEBUG:teuthology.orchestra.run.smithi135:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 2022-03-08T23:00:27.205 INFO:teuthology.orchestra.run.smithi135.stderr:10+0 records in 2022-03-08T23:00:27.206 INFO:teuthology.orchestra.run.smithi135.stderr:10+0 records out 2022-03-08T23:00:27.206 INFO:teuthology.orchestra.run.smithi135.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0146729 s, 715 MB/s 2022-03-08T23:00:27.223 INFO:tasks.kubeadm:Creating PVs + StorageClass 2022-03-08T23:00:27.224 DEBUG:tasks.kubeadm:apiVersion: v1 kind: PersistentVolume metadata: name: smithi077-lv-1 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_1 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi077 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi077-lv-2 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_2 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi077 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi077-lv-3 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_3 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi077 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi077-lv-4 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_4 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi077 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi106-lv-1 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_1 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi106 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi106-lv-2 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_2 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi106 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi106-lv-3 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_3 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi106 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi106-lv-4 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_4 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi106 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi135-lv-1 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_1 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi135 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi135-lv-2 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_2 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi135 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi135-lv-3 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_3 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi135 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi135-lv-4 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_4 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi135 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: scratch provisioner: kubernetes.io/no-provisioner volumeBindingMode: WaitForFirstConsumer 2022-03-08T23:00:27.224 DEBUG:teuthology.orchestra.run.smithi077:> kubectl create -f - 2022-03-08T23:00:27.840 INFO:teuthology.orchestra.run.smithi077.stdout:persistentvolume/smithi077-lv-1 created 2022-03-08T23:00:27.969 INFO:teuthology.orchestra.run.smithi077.stdout:persistentvolume/smithi077-lv-2 created 2022-03-08T23:00:27.972 INFO:teuthology.orchestra.run.smithi077.stdout:persistentvolume/smithi077-lv-3 created 2022-03-08T23:00:28.041 INFO:teuthology.orchestra.run.smithi077.stdout:persistentvolume/smithi077-lv-4 created 2022-03-08T23:00:28.052 INFO:teuthology.orchestra.run.smithi077.stdout:persistentvolume/smithi106-lv-1 created 2022-03-08T23:00:28.124 INFO:teuthology.orchestra.run.smithi077.stdout:persistentvolume/smithi106-lv-2 created 2022-03-08T23:00:28.128 INFO:teuthology.orchestra.run.smithi077.stdout:persistentvolume/smithi106-lv-3 created 2022-03-08T23:00:28.152 INFO:teuthology.orchestra.run.smithi077.stdout:persistentvolume/smithi106-lv-4 created 2022-03-08T23:00:28.217 INFO:teuthology.orchestra.run.smithi077.stdout:persistentvolume/smithi135-lv-1 created 2022-03-08T23:00:28.308 INFO:teuthology.orchestra.run.smithi077.stdout:persistentvolume/smithi135-lv-2 created 2022-03-08T23:00:28.329 INFO:teuthology.orchestra.run.smithi077.stdout:persistentvolume/smithi135-lv-3 created 2022-03-08T23:00:28.335 INFO:teuthology.orchestra.run.smithi077.stdout:persistentvolume/smithi135-lv-4 created 2022-03-08T23:00:28.384 INFO:teuthology.orchestra.run.smithi077.stdout:storageclass.storage.k8s.io/scratch created 2022-03-08T23:00:28.393 DEBUG:teuthology.orchestra.run.smithi077:> kubectl taint node smithi077 node-role.kubernetes.io/master- || true 2022-03-08T23:00:28.519 INFO:teuthology.orchestra.run.smithi077.stdout:node/smithi077 untainted 2022-03-08T23:00:28.523 INFO:tasks.kubeadm:Kubeadm complete, yielding 2022-03-08T23:00:28.523 INFO:teuthology.run_tasks:Running task nvme_loop... 2022-03-08T23:00:28.534 INFO:tasks.nvme_loop:Setting up nvme_loop on scratch devices... 2022-03-08T23:00:28.535 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2022-03-08T23:00:28.535 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/scratch_devs of=/dev/stdout 2022-03-08T23:00:28.542 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-03-08T23:00:28.543 DEBUG:teuthology.orchestra.run.smithi077:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2022-03-08T23:00:28.592 INFO:teuthology.orchestra.run.smithi077.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2022-03-08T23:00:28.592 INFO:teuthology.orchestra.run.smithi077.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-08T23:00:28.592 INFO:teuthology.orchestra.run.smithi077.stdout:Device: 5h/5d Inode: 502 Links: 1 2022-03-08T23:00:28.593 INFO:teuthology.orchestra.run.smithi077.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-08T23:00:28.593 INFO:teuthology.orchestra.run.smithi077.stdout:Access: 2022-03-08 23:00:26.785778031 +0000 2022-03-08T23:00:28.593 INFO:teuthology.orchestra.run.smithi077.stdout:Modify: 2022-03-08 23:00:26.785778031 +0000 2022-03-08T23:00:28.593 INFO:teuthology.orchestra.run.smithi077.stdout:Change: 2022-03-08 23:00:26.785778031 +0000 2022-03-08T23:00:28.593 INFO:teuthology.orchestra.run.smithi077.stdout: Birth: - 2022-03-08T23:00:28.598 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records in 2022-03-08T23:00:28.599 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records out 2022-03-08T23:00:28.599 INFO:teuthology.orchestra.run.smithi077.stderr:512 bytes copied, 0.000567166 s, 903 kB/s 2022-03-08T23:00:28.602 DEBUG:teuthology.orchestra.run.smithi077:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2022-03-08T23:00:28.608 INFO:teuthology.orchestra.run.smithi077.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2022-03-08T23:00:28.609 INFO:teuthology.orchestra.run.smithi077.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-08T23:00:28.609 INFO:teuthology.orchestra.run.smithi077.stdout:Device: 5h/5d Inode: 505 Links: 1 2022-03-08T23:00:28.609 INFO:teuthology.orchestra.run.smithi077.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-08T23:00:28.609 INFO:teuthology.orchestra.run.smithi077.stdout:Access: 2022-03-08 23:00:26.813778031 +0000 2022-03-08T23:00:28.609 INFO:teuthology.orchestra.run.smithi077.stdout:Modify: 2022-03-08 23:00:26.813778031 +0000 2022-03-08T23:00:28.610 INFO:teuthology.orchestra.run.smithi077.stdout:Change: 2022-03-08 23:00:26.813778031 +0000 2022-03-08T23:00:28.610 INFO:teuthology.orchestra.run.smithi077.stdout: Birth: - 2022-03-08T23:00:28.616 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records in 2022-03-08T23:00:28.616 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records out 2022-03-08T23:00:28.616 INFO:teuthology.orchestra.run.smithi077.stderr:512 bytes copied, 0.000420448 s, 1.2 MB/s 2022-03-08T23:00:28.619 DEBUG:teuthology.orchestra.run.smithi077:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2022-03-08T23:00:28.624 INFO:teuthology.orchestra.run.smithi077.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2022-03-08T23:00:28.624 INFO:teuthology.orchestra.run.smithi077.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-08T23:00:28.625 INFO:teuthology.orchestra.run.smithi077.stdout:Device: 5h/5d Inode: 512 Links: 1 2022-03-08T23:00:28.625 INFO:teuthology.orchestra.run.smithi077.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-08T23:00:28.625 INFO:teuthology.orchestra.run.smithi077.stdout:Access: 2022-03-08 23:00:26.837778032 +0000 2022-03-08T23:00:28.625 INFO:teuthology.orchestra.run.smithi077.stdout:Modify: 2022-03-08 23:00:26.837778032 +0000 2022-03-08T23:00:28.625 INFO:teuthology.orchestra.run.smithi077.stdout:Change: 2022-03-08 23:00:26.837778032 +0000 2022-03-08T23:00:28.626 INFO:teuthology.orchestra.run.smithi077.stdout: Birth: - 2022-03-08T23:00:28.635 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records in 2022-03-08T23:00:28.635 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records out 2022-03-08T23:00:28.635 INFO:teuthology.orchestra.run.smithi077.stderr:512 bytes copied, 0.00050202 s, 1.0 MB/s 2022-03-08T23:00:28.639 DEBUG:teuthology.orchestra.run.smithi077:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2022-03-08T23:00:28.687 INFO:teuthology.orchestra.run.smithi077.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2022-03-08T23:00:28.687 INFO:teuthology.orchestra.run.smithi077.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-08T23:00:28.688 INFO:teuthology.orchestra.run.smithi077.stdout:Device: 5h/5d Inode: 520 Links: 1 2022-03-08T23:00:28.688 INFO:teuthology.orchestra.run.smithi077.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-08T23:00:28.688 INFO:teuthology.orchestra.run.smithi077.stdout:Access: 2022-03-08 23:00:26.873778033 +0000 2022-03-08T23:00:28.688 INFO:teuthology.orchestra.run.smithi077.stdout:Modify: 2022-03-08 23:00:26.873778033 +0000 2022-03-08T23:00:28.688 INFO:teuthology.orchestra.run.smithi077.stdout:Change: 2022-03-08 23:00:26.873778033 +0000 2022-03-08T23:00:28.689 INFO:teuthology.orchestra.run.smithi077.stdout: Birth: - 2022-03-08T23:00:28.696 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records in 2022-03-08T23:00:28.697 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records out 2022-03-08T23:00:28.697 INFO:teuthology.orchestra.run.smithi077.stderr:512 bytes copied, 0.000530804 s, 965 kB/s 2022-03-08T23:00:28.701 DEBUG:teuthology.orchestra.run.smithi077:> sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2022-03-08T23:00:28.805 INFO:teuthology.orchestra.run.smithi077.stdout:loop 2022-03-08T23:00:28.807 INFO:tasks.nvme_loop:Connecting nvme_loop smithi077:/dev/vg_nvme/lv_1... 2022-03-08T23:00:28.808 DEBUG:teuthology.orchestra.run.smithi077:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2022-03-08T23:00:28.827 INFO:teuthology.orchestra.run.smithi077.stdout:1 2022-03-08T23:00:28.851 INFO:teuthology.orchestra.run.smithi077.stdout:/dev/vg_nvme/lv_11 2022-03-08T23:00:28.868 INFO:tasks.nvme_loop:Connecting nvme_loop smithi077:/dev/vg_nvme/lv_2... 2022-03-08T23:00:28.868 DEBUG:teuthology.orchestra.run.smithi077:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2022-03-08T23:00:28.882 INFO:teuthology.orchestra.run.smithi077.stdout:1 2022-03-08T23:00:28.906 INFO:teuthology.orchestra.run.smithi077.stdout:/dev/vg_nvme/lv_21 2022-03-08T23:00:28.927 INFO:tasks.nvme_loop:Connecting nvme_loop smithi077:/dev/vg_nvme/lv_3... 2022-03-08T23:00:28.928 DEBUG:teuthology.orchestra.run.smithi077:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2022-03-08T23:00:28.990 INFO:teuthology.orchestra.run.smithi077.stdout:1 2022-03-08T23:00:29.016 INFO:teuthology.orchestra.run.smithi077.stdout:/dev/vg_nvme/lv_31 2022-03-08T23:00:29.040 INFO:tasks.nvme_loop:Connecting nvme_loop smithi077:/dev/vg_nvme/lv_4... 2022-03-08T23:00:29.040 DEBUG:teuthology.orchestra.run.smithi077:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2022-03-08T23:00:29.058 INFO:teuthology.orchestra.run.smithi077.stdout:1 2022-03-08T23:00:29.080 INFO:teuthology.orchestra.run.smithi077.stdout:/dev/vg_nvme/lv_41 2022-03-08T23:00:29.104 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2022-03-08T23:00:29.104 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/scratch_devs of=/dev/stdout 2022-03-08T23:00:29.111 DEBUG:teuthology.orchestra.run.smithi077:> sudo nvme list 2022-03-08T23:00:29.171 INFO:teuthology.orchestra.run.smithi077.stdout:Node SN Model Namespace Usage Format FW Rev 2022-03-08T23:00:29.172 INFO:teuthology.orchestra.run.smithi077.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2022-03-08T23:00:29.172 INFO:teuthology.orchestra.run.smithi077.stdout:/dev/nvme0n1 PHFT620400ZJ400BGN INTEL SSDPEDMD400G4 1 400.09 GB / 400.09 GB 512 B + 0 B 8DV101H0 2022-03-08T23:00:29.172 INFO:teuthology.orchestra.run.smithi077.stdout:/dev/nvme1n1 85aa56076a4c3106 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-08T23:00:29.172 INFO:teuthology.orchestra.run.smithi077.stdout:/dev/nvme2n1 3fbb51d359b2bd56 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-08T23:00:29.173 INFO:teuthology.orchestra.run.smithi077.stdout:/dev/nvme3n1 8a333f193110c6ef Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-08T23:00:29.173 INFO:teuthology.orchestra.run.smithi077.stdout:/dev/nvme4n1 333c9be909c571e0 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-08T23:00:29.174 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2022-03-08T23:00:29.174 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2022-03-08T23:00:29.175 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/scratch_devs 2022-03-08T23:00:29.186 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2022-03-08T23:00:29.186 DEBUG:teuthology.orchestra.run.smithi106:> dd if=/scratch_devs of=/dev/stdout 2022-03-08T23:00:29.194 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-03-08T23:00:29.194 DEBUG:teuthology.orchestra.run.smithi106:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2022-03-08T23:00:29.201 INFO:teuthology.orchestra.run.smithi106.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2022-03-08T23:00:29.201 INFO:teuthology.orchestra.run.smithi106.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-08T23:00:29.201 INFO:teuthology.orchestra.run.smithi106.stdout:Device: 5h/5d Inode: 502 Links: 1 2022-03-08T23:00:29.202 INFO:teuthology.orchestra.run.smithi106.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-08T23:00:29.202 INFO:teuthology.orchestra.run.smithi106.stdout:Access: 2022-03-08 23:00:26.913094495 +0000 2022-03-08T23:00:29.202 INFO:teuthology.orchestra.run.smithi106.stdout:Modify: 2022-03-08 23:00:26.913094495 +0000 2022-03-08T23:00:29.202 INFO:teuthology.orchestra.run.smithi106.stdout:Change: 2022-03-08 23:00:26.913094495 +0000 2022-03-08T23:00:29.203 INFO:teuthology.orchestra.run.smithi106.stdout: Birth: - 2022-03-08T23:00:29.210 INFO:teuthology.orchestra.run.smithi106.stderr:1+0 records in 2022-03-08T23:00:29.210 INFO:teuthology.orchestra.run.smithi106.stderr:1+0 records out 2022-03-08T23:00:29.210 INFO:teuthology.orchestra.run.smithi106.stderr:512 bytes copied, 0.000586098 s, 874 kB/s 2022-03-08T23:00:29.214 DEBUG:teuthology.orchestra.run.smithi106:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2022-03-08T23:00:29.266 INFO:teuthology.orchestra.run.smithi106.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2022-03-08T23:00:29.266 INFO:teuthology.orchestra.run.smithi106.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-08T23:00:29.266 INFO:teuthology.orchestra.run.smithi106.stdout:Device: 5h/5d Inode: 506 Links: 1 2022-03-08T23:00:29.267 INFO:teuthology.orchestra.run.smithi106.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-08T23:00:29.267 INFO:teuthology.orchestra.run.smithi106.stdout:Access: 2022-03-08 23:00:26.945094494 +0000 2022-03-08T23:00:29.267 INFO:teuthology.orchestra.run.smithi106.stdout:Modify: 2022-03-08 23:00:26.945094494 +0000 2022-03-08T23:00:29.267 INFO:teuthology.orchestra.run.smithi106.stdout:Change: 2022-03-08 23:00:26.945094494 +0000 2022-03-08T23:00:29.267 INFO:teuthology.orchestra.run.smithi106.stdout: Birth: - 2022-03-08T23:00:29.275 INFO:teuthology.orchestra.run.smithi106.stderr:1+0 records in 2022-03-08T23:00:29.275 INFO:teuthology.orchestra.run.smithi106.stderr:1+0 records out 2022-03-08T23:00:29.275 INFO:teuthology.orchestra.run.smithi106.stderr:512 bytes copied, 0.000530541 s, 965 kB/s 2022-03-08T23:00:29.279 DEBUG:teuthology.orchestra.run.smithi106:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2022-03-08T23:00:29.286 INFO:teuthology.orchestra.run.smithi106.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2022-03-08T23:00:29.287 INFO:teuthology.orchestra.run.smithi106.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-08T23:00:29.287 INFO:teuthology.orchestra.run.smithi106.stdout:Device: 5h/5d Inode: 513 Links: 1 2022-03-08T23:00:29.287 INFO:teuthology.orchestra.run.smithi106.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-08T23:00:29.287 INFO:teuthology.orchestra.run.smithi106.stdout:Access: 2022-03-08 23:00:26.973094494 +0000 2022-03-08T23:00:29.287 INFO:teuthology.orchestra.run.smithi106.stdout:Modify: 2022-03-08 23:00:26.973094494 +0000 2022-03-08T23:00:29.288 INFO:teuthology.orchestra.run.smithi106.stdout:Change: 2022-03-08 23:00:26.973094494 +0000 2022-03-08T23:00:29.288 INFO:teuthology.orchestra.run.smithi106.stdout: Birth: - 2022-03-08T23:00:29.293 INFO:teuthology.orchestra.run.smithi106.stderr:1+0 records in 2022-03-08T23:00:29.294 INFO:teuthology.orchestra.run.smithi106.stderr:1+0 records out 2022-03-08T23:00:29.294 INFO:teuthology.orchestra.run.smithi106.stderr:512 bytes copied, 0.000537017 s, 953 kB/s 2022-03-08T23:00:29.298 DEBUG:teuthology.orchestra.run.smithi106:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2022-03-08T23:00:29.304 INFO:teuthology.orchestra.run.smithi106.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2022-03-08T23:00:29.305 INFO:teuthology.orchestra.run.smithi106.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-08T23:00:29.305 INFO:teuthology.orchestra.run.smithi106.stdout:Device: 5h/5d Inode: 519 Links: 1 2022-03-08T23:00:29.305 INFO:teuthology.orchestra.run.smithi106.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-08T23:00:29.305 INFO:teuthology.orchestra.run.smithi106.stdout:Access: 2022-03-08 23:00:27.049094492 +0000 2022-03-08T23:00:29.306 INFO:teuthology.orchestra.run.smithi106.stdout:Modify: 2022-03-08 23:00:27.045094492 +0000 2022-03-08T23:00:29.306 INFO:teuthology.orchestra.run.smithi106.stdout:Change: 2022-03-08 23:00:27.045094492 +0000 2022-03-08T23:00:29.306 INFO:teuthology.orchestra.run.smithi106.stdout: Birth: - 2022-03-08T23:00:29.313 INFO:teuthology.orchestra.run.smithi106.stderr:1+0 records in 2022-03-08T23:00:29.313 INFO:teuthology.orchestra.run.smithi106.stderr:1+0 records out 2022-03-08T23:00:29.313 INFO:teuthology.orchestra.run.smithi106.stderr:512 bytes copied, 0.000549141 s, 932 kB/s 2022-03-08T23:00:29.317 DEBUG:teuthology.orchestra.run.smithi106:> sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2022-03-08T23:00:29.395 INFO:teuthology.orchestra.run.smithi106.stdout:loop 2022-03-08T23:00:29.396 INFO:tasks.nvme_loop:Connecting nvme_loop smithi106:/dev/vg_nvme/lv_1... 2022-03-08T23:00:29.397 DEBUG:teuthology.orchestra.run.smithi106:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2022-03-08T23:00:29.414 INFO:teuthology.orchestra.run.smithi106.stdout:1 2022-03-08T23:00:29.433 INFO:teuthology.orchestra.run.smithi106.stdout:/dev/vg_nvme/lv_11 2022-03-08T23:00:29.450 INFO:tasks.nvme_loop:Connecting nvme_loop smithi106:/dev/vg_nvme/lv_2... 2022-03-08T23:00:29.451 DEBUG:teuthology.orchestra.run.smithi106:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2022-03-08T23:00:29.469 INFO:teuthology.orchestra.run.smithi106.stdout:1 2022-03-08T23:00:29.493 INFO:teuthology.orchestra.run.smithi106.stdout:/dev/vg_nvme/lv_21 2022-03-08T23:00:29.514 INFO:tasks.nvme_loop:Connecting nvme_loop smithi106:/dev/vg_nvme/lv_3... 2022-03-08T23:00:29.514 DEBUG:teuthology.orchestra.run.smithi106:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2022-03-08T23:00:29.533 INFO:teuthology.orchestra.run.smithi106.stdout:1 2022-03-08T23:00:29.557 INFO:teuthology.orchestra.run.smithi106.stdout:/dev/vg_nvme/lv_31 2022-03-08T23:00:29.574 INFO:tasks.nvme_loop:Connecting nvme_loop smithi106:/dev/vg_nvme/lv_4... 2022-03-08T23:00:29.575 DEBUG:teuthology.orchestra.run.smithi106:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2022-03-08T23:00:29.640 INFO:teuthology.orchestra.run.smithi106.stdout:1 2022-03-08T23:00:29.662 INFO:teuthology.orchestra.run.smithi106.stdout:/dev/vg_nvme/lv_41 2022-03-08T23:00:29.682 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2022-03-08T23:00:29.683 DEBUG:teuthology.orchestra.run.smithi106:> dd if=/scratch_devs of=/dev/stdout 2022-03-08T23:00:29.688 DEBUG:teuthology.orchestra.run.smithi106:> sudo nvme list 2022-03-08T23:00:29.699 INFO:teuthology.orchestra.run.smithi106.stdout:Node SN Model Namespace Usage Format FW Rev 2022-03-08T23:00:29.699 INFO:teuthology.orchestra.run.smithi106.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2022-03-08T23:00:29.699 INFO:teuthology.orchestra.run.smithi106.stdout:/dev/nvme0n1 PHFT6204013R400BGN INTEL SSDPEDMD400G4 1 400.09 GB / 400.09 GB 512 B + 0 B 8DV101H0 2022-03-08T23:00:29.699 INFO:teuthology.orchestra.run.smithi106.stdout:/dev/nvme1n1 c7e5f58956341399 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-08T23:00:29.700 INFO:teuthology.orchestra.run.smithi106.stdout:/dev/nvme2n1 a044a5bb6a964722 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-08T23:00:29.700 INFO:teuthology.orchestra.run.smithi106.stdout:/dev/nvme3n1 263d6feb84d58182 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-08T23:00:29.701 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1'] 2022-03-08T23:00:30.702 DEBUG:teuthology.orchestra.run.smithi106:> sudo nvme list 2022-03-08T23:00:30.717 INFO:teuthology.orchestra.run.smithi106.stdout:Node SN Model Namespace Usage Format FW Rev 2022-03-08T23:00:30.718 INFO:teuthology.orchestra.run.smithi106.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2022-03-08T23:00:30.718 INFO:teuthology.orchestra.run.smithi106.stdout:/dev/nvme0n1 PHFT6204013R400BGN INTEL SSDPEDMD400G4 1 400.09 GB / 400.09 GB 512 B + 0 B 8DV101H0 2022-03-08T23:00:30.718 INFO:teuthology.orchestra.run.smithi106.stdout:/dev/nvme1n1 c7e5f58956341399 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-08T23:00:30.718 INFO:teuthology.orchestra.run.smithi106.stdout:/dev/nvme2n1 a044a5bb6a964722 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-08T23:00:30.719 INFO:teuthology.orchestra.run.smithi106.stdout:/dev/nvme3n1 263d6feb84d58182 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-08T23:00:30.719 INFO:teuthology.orchestra.run.smithi106.stdout:/dev/nvme4n1 25f25f0d4ffc9199 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-08T23:00:30.720 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2022-03-08T23:00:30.720 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2022-03-08T23:00:30.721 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd of=/scratch_devs 2022-03-08T23:00:30.782 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2022-03-08T23:00:30.782 DEBUG:teuthology.orchestra.run.smithi135:> dd if=/scratch_devs of=/dev/stdout 2022-03-08T23:00:30.788 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-03-08T23:00:30.788 DEBUG:teuthology.orchestra.run.smithi135:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2022-03-08T23:00:30.793 INFO:teuthology.orchestra.run.smithi135.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2022-03-08T23:00:30.793 INFO:teuthology.orchestra.run.smithi135.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-08T23:00:30.793 INFO:teuthology.orchestra.run.smithi135.stdout:Device: 5h/5d Inode: 496 Links: 1 2022-03-08T23:00:30.794 INFO:teuthology.orchestra.run.smithi135.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-08T23:00:30.794 INFO:teuthology.orchestra.run.smithi135.stdout:Access: 2022-03-08 23:00:27.080468833 +0000 2022-03-08T23:00:30.794 INFO:teuthology.orchestra.run.smithi135.stdout:Modify: 2022-03-08 23:00:27.080468833 +0000 2022-03-08T23:00:30.794 INFO:teuthology.orchestra.run.smithi135.stdout:Change: 2022-03-08 23:00:27.080468833 +0000 2022-03-08T23:00:30.795 INFO:teuthology.orchestra.run.smithi135.stdout: Birth: - 2022-03-08T23:00:30.798 INFO:teuthology.orchestra.run.smithi135.stderr:1+0 records in 2022-03-08T23:00:30.799 INFO:teuthology.orchestra.run.smithi135.stderr:1+0 records out 2022-03-08T23:00:30.799 INFO:teuthology.orchestra.run.smithi135.stderr:512 bytes copied, 0.000462758 s, 1.1 MB/s 2022-03-08T23:00:30.802 DEBUG:teuthology.orchestra.run.smithi135:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2022-03-08T23:00:30.807 INFO:teuthology.orchestra.run.smithi135.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2022-03-08T23:00:30.807 INFO:teuthology.orchestra.run.smithi135.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-08T23:00:30.807 INFO:teuthology.orchestra.run.smithi135.stdout:Device: 5h/5d Inode: 502 Links: 1 2022-03-08T23:00:30.807 INFO:teuthology.orchestra.run.smithi135.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-08T23:00:30.808 INFO:teuthology.orchestra.run.smithi135.stdout:Access: 2022-03-08 23:00:27.152468836 +0000 2022-03-08T23:00:30.808 INFO:teuthology.orchestra.run.smithi135.stdout:Modify: 2022-03-08 23:00:27.152468836 +0000 2022-03-08T23:00:30.808 INFO:teuthology.orchestra.run.smithi135.stdout:Change: 2022-03-08 23:00:27.152468836 +0000 2022-03-08T23:00:30.808 INFO:teuthology.orchestra.run.smithi135.stdout: Birth: - 2022-03-08T23:00:30.812 INFO:teuthology.orchestra.run.smithi135.stderr:1+0 records in 2022-03-08T23:00:30.813 INFO:teuthology.orchestra.run.smithi135.stderr:1+0 records out 2022-03-08T23:00:30.813 INFO:teuthology.orchestra.run.smithi135.stderr:512 bytes copied, 0.000415655 s, 1.2 MB/s 2022-03-08T23:00:30.817 DEBUG:teuthology.orchestra.run.smithi135:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2022-03-08T23:00:30.822 INFO:teuthology.orchestra.run.smithi135.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2022-03-08T23:00:30.822 INFO:teuthology.orchestra.run.smithi135.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-08T23:00:30.823 INFO:teuthology.orchestra.run.smithi135.stdout:Device: 5h/5d Inode: 507 Links: 1 2022-03-08T23:00:30.823 INFO:teuthology.orchestra.run.smithi135.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-08T23:00:30.823 INFO:teuthology.orchestra.run.smithi135.stdout:Access: 2022-03-08 23:00:27.180468836 +0000 2022-03-08T23:00:30.823 INFO:teuthology.orchestra.run.smithi135.stdout:Modify: 2022-03-08 23:00:27.180468836 +0000 2022-03-08T23:00:30.824 INFO:teuthology.orchestra.run.smithi135.stdout:Change: 2022-03-08 23:00:27.180468836 +0000 2022-03-08T23:00:30.824 INFO:teuthology.orchestra.run.smithi135.stdout: Birth: - 2022-03-08T23:00:30.829 INFO:teuthology.orchestra.run.smithi135.stderr:1+0 records in 2022-03-08T23:00:30.829 INFO:teuthology.orchestra.run.smithi135.stderr:1+0 records out 2022-03-08T23:00:30.829 INFO:teuthology.orchestra.run.smithi135.stderr:512 bytes copied, 0.000513356 s, 997 kB/s 2022-03-08T23:00:30.833 DEBUG:teuthology.orchestra.run.smithi135:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2022-03-08T23:00:30.838 INFO:teuthology.orchestra.run.smithi135.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2022-03-08T23:00:30.838 INFO:teuthology.orchestra.run.smithi135.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-08T23:00:30.839 INFO:teuthology.orchestra.run.smithi135.stdout:Device: 5h/5d Inode: 514 Links: 1 2022-03-08T23:00:30.839 INFO:teuthology.orchestra.run.smithi135.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-08T23:00:30.839 INFO:teuthology.orchestra.run.smithi135.stdout:Access: 2022-03-08 23:00:27.208468837 +0000 2022-03-08T23:00:30.839 INFO:teuthology.orchestra.run.smithi135.stdout:Modify: 2022-03-08 23:00:27.208468837 +0000 2022-03-08T23:00:30.839 INFO:teuthology.orchestra.run.smithi135.stdout:Change: 2022-03-08 23:00:27.208468837 +0000 2022-03-08T23:00:30.840 INFO:teuthology.orchestra.run.smithi135.stdout: Birth: - 2022-03-08T23:00:30.847 INFO:teuthology.orchestra.run.smithi135.stderr:1+0 records in 2022-03-08T23:00:30.847 INFO:teuthology.orchestra.run.smithi135.stderr:1+0 records out 2022-03-08T23:00:30.847 INFO:teuthology.orchestra.run.smithi135.stderr:512 bytes copied, 0.000532164 s, 962 kB/s 2022-03-08T23:00:30.851 DEBUG:teuthology.orchestra.run.smithi135:> sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2022-03-08T23:00:31.247 INFO:teuthology.orchestra.run.smithi135.stdout:loop 2022-03-08T23:00:31.248 INFO:tasks.nvme_loop:Connecting nvme_loop smithi135:/dev/vg_nvme/lv_1... 2022-03-08T23:00:31.248 DEBUG:teuthology.orchestra.run.smithi135:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2022-03-08T23:00:31.269 INFO:teuthology.orchestra.run.smithi135.stdout:1 2022-03-08T23:00:31.286 INFO:teuthology.orchestra.run.smithi135.stdout:/dev/vg_nvme/lv_11 2022-03-08T23:00:31.309 INFO:tasks.nvme_loop:Connecting nvme_loop smithi135:/dev/vg_nvme/lv_2... 2022-03-08T23:00:31.310 DEBUG:teuthology.orchestra.run.smithi135:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2022-03-08T23:00:31.330 INFO:teuthology.orchestra.run.smithi135.stdout:1 2022-03-08T23:00:31.349 INFO:teuthology.orchestra.run.smithi135.stdout:/dev/vg_nvme/lv_21 2022-03-08T23:00:31.368 INFO:tasks.nvme_loop:Connecting nvme_loop smithi135:/dev/vg_nvme/lv_3... 2022-03-08T23:00:31.369 DEBUG:teuthology.orchestra.run.smithi135:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2022-03-08T23:00:31.432 INFO:teuthology.orchestra.run.smithi135.stdout:1 2022-03-08T23:00:31.456 INFO:teuthology.orchestra.run.smithi135.stdout:/dev/vg_nvme/lv_31 2022-03-08T23:00:31.475 INFO:tasks.nvme_loop:Connecting nvme_loop smithi135:/dev/vg_nvme/lv_4... 2022-03-08T23:00:31.475 DEBUG:teuthology.orchestra.run.smithi135:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2022-03-08T23:00:31.493 INFO:teuthology.orchestra.run.smithi135.stdout:1 2022-03-08T23:00:31.522 INFO:teuthology.orchestra.run.smithi135.stdout:/dev/vg_nvme/lv_41 2022-03-08T23:00:31.539 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2022-03-08T23:00:31.539 DEBUG:teuthology.orchestra.run.smithi135:> dd if=/scratch_devs of=/dev/stdout 2022-03-08T23:00:31.544 DEBUG:teuthology.orchestra.run.smithi135:> sudo nvme list 2022-03-08T23:00:31.593 INFO:teuthology.orchestra.run.smithi135.stdout:Node SN Model Namespace Usage Format FW Rev 2022-03-08T23:00:31.594 INFO:teuthology.orchestra.run.smithi135.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2022-03-08T23:00:31.594 INFO:teuthology.orchestra.run.smithi135.stdout:/dev/nvme0n1 PHFT620401BA400BGN INTEL SSDPEDMD400G4 1 400.09 GB / 400.09 GB 512 B + 0 B 8DV101H0 2022-03-08T23:00:31.594 INFO:teuthology.orchestra.run.smithi135.stdout:/dev/nvme1n1 1e4320bd92dea7f0 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-08T23:00:31.594 INFO:teuthology.orchestra.run.smithi135.stdout:/dev/nvme2n1 7d3b97598718dcc2 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-08T23:00:31.595 INFO:teuthology.orchestra.run.smithi135.stdout:/dev/nvme3n1 7b51f60e3c8e49f4 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-08T23:00:31.595 INFO:teuthology.orchestra.run.smithi135.stdout:/dev/nvme4n1 cc2722a882850eaf Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-08T23:00:31.596 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2022-03-08T23:00:31.596 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2022-03-08T23:00:31.596 DEBUG:teuthology.orchestra.run.smithi135:> sudo dd of=/scratch_devs 2022-03-08T23:00:31.607 INFO:teuthology.run_tasks:Running task rook... 2022-03-08T23:00:31.669 INFO:tasks.rook:Rook start 2022-03-08T23:00:31.670 INFO:tasks.rook:Config: {'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '7a44e9ebca556862e9704be295a07719d7db47b3', 'rook_image': 'rook/ceph:master'} 2022-03-08T23:00:31.670 INFO:tasks.rook:Ceph image is quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3 2022-03-08T23:00:31.671 INFO:tasks.rook:Cloning https://github.com/rook/rook branch master 2022-03-08T23:00:31.671 DEBUG:teuthology.orchestra.run.smithi077:> rm -rf rook && git clone --single-branch --branch master https://github.com/rook/rook rook 2022-03-08T23:00:31.678 INFO:teuthology.orchestra.run.smithi077.stderr:Cloning into 'rook'... 2022-03-08T23:00:34.611 INFO:tasks.rook:/home/teuthworker 2022-03-08T23:00:34.611 DEBUG:teuthology.orchestra.run.smithi077:> PATH=/sbin:/usr/sbin ip addr show 2022-03-08T23:00:34.663 INFO:teuthology.orchestra.run.smithi077.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2022-03-08T23:00:34.663 INFO:teuthology.orchestra.run.smithi077.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2022-03-08T23:00:34.664 INFO:teuthology.orchestra.run.smithi077.stdout: inet 127.0.0.1/8 scope host lo 2022-03-08T23:00:34.664 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2022-03-08T23:00:34.664 INFO:teuthology.orchestra.run.smithi077.stdout: inet6 ::1/128 scope host 2022-03-08T23:00:34.664 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2022-03-08T23:00:34.665 INFO:teuthology.orchestra.run.smithi077.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2022-03-08T23:00:34.665 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 0c:c4:7a:d8:d9:d8 brd ff:ff:ff:ff:ff:ff 2022-03-08T23:00:34.665 INFO:teuthology.orchestra.run.smithi077.stdout: altname enp7s0f0 2022-03-08T23:00:34.665 INFO:teuthology.orchestra.run.smithi077.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2022-03-08T23:00:34.665 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 0c:c4:7a:d8:d9:d9 brd ff:ff:ff:ff:ff:ff 2022-03-08T23:00:34.666 INFO:teuthology.orchestra.run.smithi077.stdout: altname enp7s0f1 2022-03-08T23:00:34.666 INFO:teuthology.orchestra.run.smithi077.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2022-03-08T23:00:34.666 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 0c:c4:7a:8f:d3:d2 brd ff:ff:ff:ff:ff:ff 2022-03-08T23:00:34.666 INFO:teuthology.orchestra.run.smithi077.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2022-03-08T23:00:34.667 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 0c:c4:7a:8f:d3:d3 brd ff:ff:ff:ff:ff:ff 2022-03-08T23:00:34.667 INFO:teuthology.orchestra.run.smithi077.stdout: inet 172.21.15.77/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2022-03-08T23:00:34.667 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft 42857sec preferred_lft 42857sec 2022-03-08T23:00:34.667 INFO:teuthology.orchestra.run.smithi077.stdout: inet6 fe80::ec4:7aff:fe8f:d3d3/64 scope link 2022-03-08T23:00:34.668 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2022-03-08T23:00:34.668 INFO:teuthology.orchestra.run.smithi077.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2022-03-08T23:00:34.668 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 02:42:82:73:e9:f3 brd ff:ff:ff:ff:ff:ff 2022-03-08T23:00:34.668 INFO:teuthology.orchestra.run.smithi077.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2022-03-08T23:00:34.668 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2022-03-08T23:00:34.670 DEBUG:teuthology.orchestra.run.smithi077:> which systemctl 2022-03-08T23:00:34.675 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/systemctl 2022-03-08T23:00:34.676 DEBUG:teuthology.orchestra.run.smithi077:> test -f /run/.containerenv -o -f /.dockerenv 2022-03-08T23:00:34.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-03-08T23:00:34.680 DEBUG:teuthology.orchestra.run.smithi077:> true 2022-03-08T23:00:34.751 DEBUG:teuthology.orchestra.run.smithi077:> sudo lsb_release -is 2022-03-08T23:00:34.793 INFO:teuthology.orchestra.run.smithi077.stdout:Ubuntu 2022-03-08T23:00:34.799 DEBUG:teuthology.misc:System to be installed: Ubuntu 2022-03-08T23:00:34.799 INFO:tasks.rook:['__class__', '__del__', '__delattr__', '__dir__', '__eq__', '__format__', '__ge__', '__getattribute__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__le__', '__lt__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '_format_size', '_reconnect', '_runner', '_set_iface_and_cidr', '_sftp_get_file', '_sftp_get_size', '_sftp_open_file', '_sftp_put_file', 'chcon', 'chmod', 'connect', 'copy_file', 'ensure_online', 'get_file', 'get_tar', 'get_tar_stream', 'mkdtemp', 'mktemp', 'move_file', 'put_file', 'read_file', 'reconnect', 'remove', 'run', 'sh', 'sh_file', 'sudo_write_file', 'write_file'] 2022-03-08T23:00:34.812 DEBUG:teuthology.orchestra.remote:smithi077:rook/deploy/examples/operator.yaml is 20KB 2022-03-08T23:00:34.824 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2022-03-08T23:00:34.825 DEBUG:teuthology.orchestra.run.smithi077:> dd if=rook/deploy/examples/operator.yaml of=/dev/stdout 2022-03-08T23:00:34.834 INFO:tasks.rook:Patching operator to use image rook/ceph:master 2022-03-08T23:00:34.874 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2022-03-08T23:00:34.874 DEBUG:teuthology.orchestra.run.smithi077:> dd of=operator.yaml 2022-03-08T23:00:34.883 INFO:tasks.rook:Deploying operator 2022-03-08T23:00:34.893 DEBUG:teuthology.orchestra.remote:smithi077:rook/deploy/examples/operator.yaml is 20KB 2022-03-08T23:00:34.913 DEBUG:teuthology.orchestra.remote:smithi077:rook/deploy/examples/operator.yaml is 20KB 2022-03-08T23:00:34.926 DEBUG:teuthology.orchestra.run.smithi077:> kubectl create -f rook/deploy/examples/crds.yaml -f rook/deploy/examples/common.yaml -f operator.yaml 2022-03-08T23:00:35.135 INFO:teuthology.orchestra.run.smithi077.stdout:customresourcedefinition.apiextensions.k8s.io/cephblockpools.ceph.rook.io created 2022-03-08T23:00:35.220 INFO:teuthology.orchestra.run.smithi077.stdout:customresourcedefinition.apiextensions.k8s.io/cephbucketnotifications.ceph.rook.io created 2022-03-08T23:00:35.246 INFO:teuthology.orchestra.run.smithi077.stdout:customresourcedefinition.apiextensions.k8s.io/cephbuckettopics.ceph.rook.io created 2022-03-08T23:00:35.304 INFO:teuthology.orchestra.run.smithi077.stdout:customresourcedefinition.apiextensions.k8s.io/cephclients.ceph.rook.io created 2022-03-08T23:00:35.439 INFO:teuthology.orchestra.run.smithi077.stdout:customresourcedefinition.apiextensions.k8s.io/cephclusters.ceph.rook.io created 2022-03-08T23:00:35.472 INFO:teuthology.orchestra.run.smithi077.stdout:customresourcedefinition.apiextensions.k8s.io/cephfilesystemmirrors.ceph.rook.io created 2022-03-08T23:00:35.596 INFO:teuthology.orchestra.run.smithi077.stdout:customresourcedefinition.apiextensions.k8s.io/cephfilesystems.ceph.rook.io created 2022-03-08T23:00:35.601 INFO:teuthology.orchestra.run.smithi077.stdout:customresourcedefinition.apiextensions.k8s.io/cephfilesystemsubvolumegroups.ceph.rook.io created 2022-03-08T23:00:35.642 INFO:teuthology.orchestra.run.smithi077.stdout:customresourcedefinition.apiextensions.k8s.io/cephnfses.ceph.rook.io created 2022-03-08T23:00:35.773 INFO:teuthology.orchestra.run.smithi077.stdout:customresourcedefinition.apiextensions.k8s.io/cephobjectrealms.ceph.rook.io created 2022-03-08T23:00:35.844 INFO:teuthology.orchestra.run.smithi077.stdout:customresourcedefinition.apiextensions.k8s.io/cephobjectstores.ceph.rook.io created 2022-03-08T23:00:35.951 INFO:teuthology.orchestra.run.smithi077.stdout:customresourcedefinition.apiextensions.k8s.io/cephobjectstoreusers.ceph.rook.io created 2022-03-08T23:00:35.975 INFO:teuthology.orchestra.run.smithi077.stdout:customresourcedefinition.apiextensions.k8s.io/cephobjectzonegroups.ceph.rook.io created 2022-03-08T23:00:36.005 INFO:teuthology.orchestra.run.smithi077.stdout:customresourcedefinition.apiextensions.k8s.io/cephobjectzones.ceph.rook.io created 2022-03-08T23:00:36.030 INFO:teuthology.orchestra.run.smithi077.stdout:customresourcedefinition.apiextensions.k8s.io/cephrbdmirrors.ceph.rook.io created 2022-03-08T23:00:36.164 INFO:teuthology.orchestra.run.smithi077.stdout:customresourcedefinition.apiextensions.k8s.io/objectbucketclaims.objectbucket.io created 2022-03-08T23:00:36.176 INFO:teuthology.orchestra.run.smithi077.stdout:customresourcedefinition.apiextensions.k8s.io/objectbuckets.objectbucket.io created 2022-03-08T23:00:36.345 INFO:teuthology.orchestra.run.smithi077.stdout:namespace/rook-ceph created 2022-03-08T23:00:36.395 INFO:teuthology.orchestra.run.smithi077.stdout:clusterrole.rbac.authorization.k8s.io/cephfs-csi-nodeplugin created 2022-03-08T23:00:36.420 INFO:teuthology.orchestra.run.smithi077.stdout:clusterrole.rbac.authorization.k8s.io/cephfs-external-provisioner-runner created 2022-03-08T23:00:36.549 INFO:teuthology.orchestra.run.smithi077.stdout:clusterrole.rbac.authorization.k8s.io/psp:rook created 2022-03-08T23:00:36.580 INFO:teuthology.orchestra.run.smithi077.stdout:clusterrole.rbac.authorization.k8s.io/rbd-csi-nodeplugin created 2022-03-08T23:00:36.627 INFO:teuthology.orchestra.run.smithi077.stdout:clusterrole.rbac.authorization.k8s.io/rbd-external-provisioner-runner created 2022-03-08T23:00:36.705 INFO:teuthology.orchestra.run.smithi077.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-cluster-mgmt created 2022-03-08T23:00:36.712 INFO:teuthology.orchestra.run.smithi077.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-global created 2022-03-08T23:00:36.767 INFO:teuthology.orchestra.run.smithi077.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-mgr-cluster created 2022-03-08T23:00:36.780 INFO:teuthology.orchestra.run.smithi077.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-mgr-system created 2022-03-08T23:00:36.793 INFO:teuthology.orchestra.run.smithi077.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-object-bucket created 2022-03-08T23:00:36.835 INFO:teuthology.orchestra.run.smithi077.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-osd created 2022-03-08T23:00:36.838 INFO:teuthology.orchestra.run.smithi077.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-system created 2022-03-08T23:00:36.873 INFO:teuthology.orchestra.run.smithi077.stdout:clusterrolebinding.rbac.authorization.k8s.io/cephfs-csi-nodeplugin created 2022-03-08T23:00:36.893 INFO:teuthology.orchestra.run.smithi077.stdout:clusterrolebinding.rbac.authorization.k8s.io/cephfs-csi-provisioner-role created 2022-03-08T23:00:36.910 INFO:teuthology.orchestra.run.smithi077.stdout:clusterrolebinding.rbac.authorization.k8s.io/rbd-csi-nodeplugin created 2022-03-08T23:00:36.928 INFO:teuthology.orchestra.run.smithi077.stdout:clusterrolebinding.rbac.authorization.k8s.io/rbd-csi-provisioner-role created 2022-03-08T23:00:36.998 INFO:teuthology.orchestra.run.smithi077.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-global created 2022-03-08T23:00:37.014 INFO:teuthology.orchestra.run.smithi077.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-mgr-cluster created 2022-03-08T23:00:37.026 INFO:teuthology.orchestra.run.smithi077.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-object-bucket created 2022-03-08T23:00:37.035 INFO:teuthology.orchestra.run.smithi077.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-osd created 2022-03-08T23:00:37.051 INFO:teuthology.orchestra.run.smithi077.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-system created 2022-03-08T23:00:37.289 INFO:teuthology.orchestra.run.smithi077.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-system-psp created 2022-03-08T23:00:37.327 INFO:teuthology.orchestra.run.smithi077.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-csi-cephfs-plugin-sa-psp created 2022-03-08T23:00:37.505 INFO:teuthology.orchestra.run.smithi077.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-csi-cephfs-provisioner-sa-psp created 2022-03-08T23:00:37.508 INFO:teuthology.orchestra.run.smithi077.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-csi-rbd-plugin-sa-psp created 2022-03-08T23:00:37.551 INFO:teuthology.orchestra.run.smithi077.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-csi-rbd-provisioner-sa-psp created 2022-03-08T23:00:37.577 INFO:teuthology.orchestra.run.smithi077.stderr:Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+ 2022-03-08T23:00:37.577 INFO:teuthology.orchestra.run.smithi077.stdout:podsecuritypolicy.policy/00-rook-privileged created 2022-03-08T23:00:37.844 INFO:teuthology.orchestra.run.smithi077.stdout:role.rbac.authorization.k8s.io/cephfs-external-provisioner-cfg created 2022-03-08T23:00:37.846 INFO:teuthology.orchestra.run.smithi077.stdout:role.rbac.authorization.k8s.io/rbd-csi-nodeplugin created 2022-03-08T23:00:38.194 INFO:teuthology.orchestra.run.smithi077.stdout:role.rbac.authorization.k8s.io/rbd-external-provisioner-cfg created 2022-03-08T23:00:38.264 INFO:teuthology.orchestra.run.smithi077.stdout:role.rbac.authorization.k8s.io/rook-ceph-cmd-reporter created 2022-03-08T23:00:38.397 INFO:teuthology.orchestra.run.smithi077.stdout:role.rbac.authorization.k8s.io/rook-ceph-mgr created 2022-03-08T23:00:38.423 INFO:teuthology.orchestra.run.smithi077.stdout:role.rbac.authorization.k8s.io/rook-ceph-osd created 2022-03-08T23:00:38.438 INFO:teuthology.orchestra.run.smithi077.stdout:role.rbac.authorization.k8s.io/rook-ceph-purge-osd created 2022-03-08T23:00:38.456 INFO:teuthology.orchestra.run.smithi077.stdout:role.rbac.authorization.k8s.io/rook-ceph-system created 2022-03-08T23:00:38.469 INFO:teuthology.orchestra.run.smithi077.stdout:rolebinding.rbac.authorization.k8s.io/cephfs-csi-provisioner-role-cfg created 2022-03-08T23:00:38.583 INFO:teuthology.orchestra.run.smithi077.stdout:rolebinding.rbac.authorization.k8s.io/rbd-csi-nodeplugin-role-cfg created 2022-03-08T23:00:38.588 INFO:teuthology.orchestra.run.smithi077.stdout:rolebinding.rbac.authorization.k8s.io/rbd-csi-provisioner-role-cfg created 2022-03-08T23:00:38.652 INFO:teuthology.orchestra.run.smithi077.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-cluster-mgmt created 2022-03-08T23:00:38.723 INFO:teuthology.orchestra.run.smithi077.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-cmd-reporter created 2022-03-08T23:00:38.749 INFO:teuthology.orchestra.run.smithi077.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-cmd-reporter-psp created 2022-03-08T23:00:38.759 INFO:teuthology.orchestra.run.smithi077.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-default-psp created 2022-03-08T23:00:38.771 INFO:teuthology.orchestra.run.smithi077.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-mgr created 2022-03-08T23:00:38.802 INFO:teuthology.orchestra.run.smithi077.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-mgr-psp created 2022-03-08T23:00:38.818 INFO:teuthology.orchestra.run.smithi077.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-mgr-system created 2022-03-08T23:00:38.875 INFO:teuthology.orchestra.run.smithi077.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-osd created 2022-03-08T23:00:38.879 INFO:teuthology.orchestra.run.smithi077.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-osd-psp created 2022-03-08T23:00:38.884 INFO:teuthology.orchestra.run.smithi077.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-purge-osd created 2022-03-08T23:00:38.894 INFO:teuthology.orchestra.run.smithi077.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-system created 2022-03-08T23:00:38.905 INFO:teuthology.orchestra.run.smithi077.stdout:serviceaccount/rook-ceph-cmd-reporter created 2022-03-08T23:00:38.915 INFO:teuthology.orchestra.run.smithi077.stdout:serviceaccount/rook-ceph-mgr created 2022-03-08T23:00:38.942 INFO:teuthology.orchestra.run.smithi077.stdout:serviceaccount/rook-ceph-osd created 2022-03-08T23:00:38.966 INFO:teuthology.orchestra.run.smithi077.stdout:serviceaccount/rook-ceph-purge-osd created 2022-03-08T23:00:39.051 INFO:teuthology.orchestra.run.smithi077.stdout:serviceaccount/rook-ceph-system created 2022-03-08T23:00:39.055 INFO:teuthology.orchestra.run.smithi077.stdout:serviceaccount/rook-csi-cephfs-plugin-sa created 2022-03-08T23:00:39.086 INFO:teuthology.orchestra.run.smithi077.stdout:serviceaccount/rook-csi-cephfs-provisioner-sa created 2022-03-08T23:00:39.148 INFO:teuthology.orchestra.run.smithi077.stdout:serviceaccount/rook-csi-rbd-plugin-sa created 2022-03-08T23:00:39.447 INFO:teuthology.orchestra.run.smithi077.stdout:serviceaccount/rook-csi-rbd-provisioner-sa created 2022-03-08T23:00:39.469 INFO:teuthology.orchestra.run.smithi077.stdout:configmap/rook-ceph-operator-config created 2022-03-08T23:00:39.629 INFO:teuthology.orchestra.run.smithi077.stdout:deployment.apps/rook-ceph-operator created 2022-03-08T23:00:39.632 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-08T23:00:39.930 INFO:teuthology.orchestra.run.smithi077.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T23:00:39.931 INFO:teuthology.orchestra.run.smithi077.stdout:rook-ceph-operator-75dd789779-dfk8z 0/1 Pending 0 0s 2022-03-08T23:00:49.933 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-08T23:00:50.002 INFO:teuthology.orchestra.run.smithi077.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T23:00:50.003 INFO:teuthology.orchestra.run.smithi077.stdout:rook-ceph-operator-75dd789779-dfk8z 0/1 ContainerCreating 0 11s 2022-03-08T23:01:00.008 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-08T23:01:00.073 INFO:teuthology.orchestra.run.smithi077.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T23:01:00.073 INFO:teuthology.orchestra.run.smithi077.stdout:rook-ceph-operator-75dd789779-dfk8z 0/1 ContainerCreating 0 21s 2022-03-08T23:01:10.076 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-08T23:01:10.137 INFO:teuthology.orchestra.run.smithi077.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T23:01:10.138 INFO:teuthology.orchestra.run.smithi077.stdout:rook-ceph-operator-75dd789779-dfk8z 0/1 ContainerCreating 0 31s 2022-03-08T23:01:20.141 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-08T23:01:20.202 INFO:teuthology.orchestra.run.smithi077.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T23:01:20.203 INFO:teuthology.orchestra.run.smithi077.stdout:rook-ceph-operator-75dd789779-dfk8z 1/1 Running 0 41s 2022-03-08T23:01:20.205 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph logs -f rook-ceph-operator-75dd789779-dfk8z 2022-03-08T23:01:20.213 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2022-03-08T23:01:20.213 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/scratch_devs of=/dev/stdout 2022-03-08T23:01:20.262 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2022-03-08T23:01:20.262 DEBUG:teuthology.orchestra.run.smithi106:> dd if=/scratch_devs of=/dev/stdout 2022-03-08T23:01:20.270 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2022-03-08T23:01:20.271 DEBUG:teuthology.orchestra.run.smithi135:> dd if=/scratch_devs of=/dev/stdout 2022-03-08T23:01:20.279 INFO:tasks.rook: override: [mgr] debug mgr = 20 2022-03-08T23:01:20.279 INFO:tasks.rook: override: [mgr] debug ms = 1 2022-03-08T23:01:20.279 INFO:tasks.rook: override: [mon] debug mon = 20 2022-03-08T23:01:20.280 INFO:tasks.rook: override: [mon] debug ms = 1 2022-03-08T23:01:20.280 INFO:tasks.rook: override: [mon] debug paxos = 20 2022-03-08T23:01:20.280 INFO:tasks.rook: override: [osd] debug ms = 1 2022-03-08T23:01:20.281 INFO:tasks.rook: override: [osd] debug osd = 20 2022-03-08T23:01:20.281 INFO:tasks.rook:Config: b"[global]\n\nlog to file = true\n\nmon clock drift allowed = 1.000\n\n# replicate across OSDs, not hosts\nosd crush chooseleaf type = 0\n\n# enable some debugging\nauth debug = true\nms die on old message = true\nms die on bug = true\ndebug asserts on shutdown = true\n\n\n[osd]\n# debugging\nosd debug shutdown = true\nosd debug op order = true\nosd debug verify stray on activate = true\nosd debug pg log writeout = true\nosd debug verify cached snaps = true\nosd debug verify missing on start = true\nosd debug misdirected ops = true\nosd op queue = debug_random\nosd op queue cut off = debug_random\nosd shutdown pgref assert = true\nbdev debug aio = true\nosd sloppy crc = true\ndebug ms = 1\ndebug osd = 20\n\n\n[mon]\n# rotate auth tickets quickly to exercise renewal paths\nauth mon ticket ttl = 660# 11m\nauth service ticket ttl = 240# 4m\n\n# don't complain about global id reclaim\nmon_warn_on_insecure_global_id_reclaim = false\nmon_warn_on_insecure_global_id_reclaim_allowed = false\ndebug mon = 20\ndebug ms = 1\ndebug paxos = 20\n[mgr]\ndebug mgr = 20\ndebug ms = 1\n" 2022-03-08T23:01:20.284 DEBUG:teuthology.orchestra.run.smithi077:> kubectl create -f - 2022-03-08T23:01:20.318 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:01:14.213137 I | rookcmd: starting Rook v1.8.0-alpha.0.381.g235781e28 with arguments '/usr/local/bin/rook ceph operator' 2022-03-08T23:01:20.319 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:01:14.213313 I | rookcmd: flag values: --enable-machine-disruption-budget=false, --help=false, --kubeconfig=, --log-level=INFO, --operator-image=, --service-account= 2022-03-08T23:01:20.319 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:01:14.213320 I | cephcmd: starting Rook-Ceph operator 2022-03-08T23:01:20.319 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:01:14.372316 I | cephcmd: base ceph version inside the rook operator image is "ceph version 16.2.7 (dd0603118f56ab514f133c8d2e3adfc983942503) pacific (stable)" 2022-03-08T23:01:20.319 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:01:14.380433 I | op-k8sutil: ROOK_CURRENT_NAMESPACE_ONLY="false" (env var) 2022-03-08T23:01:20.320 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:01:14.380457 I | operator: watching all namespaces for Ceph CRs 2022-03-08T23:01:20.320 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:01:14.380514 I | operator: setting up schemes 2022-03-08T23:01:20.320 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:01:14.384419 I | operator: setting up the controller-runtime manager 2022-03-08T23:01:20.320 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:01:14.588905 I | operator: looking for admission webhook secret "rook-ceph-admission-controller" 2022-03-08T23:01:20.321 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:01:14.607140 I | operator: admission webhook secret "rook-ceph-admission-controller" not found. proceeding without the admission controller 2022-03-08T23:01:20.321 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:01:14.607214 I | ceph-cluster-controller: successfully started 2022-03-08T23:01:20.321 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:01:14.607303 I | ceph-cluster-controller: enabling hotplug orchestration 2022-03-08T23:01:20.321 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:01:14.607380 I | ceph-crashcollector-controller: successfully started 2022-03-08T23:01:20.321 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:01:14.607416 I | ceph-block-pool-controller: successfully started 2022-03-08T23:01:20.322 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:01:14.607477 I | ceph-object-store-user-controller: successfully started 2022-03-08T23:01:20.322 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:01:14.607513 I | ceph-object-realm-controller: successfully started 2022-03-08T23:01:20.322 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:01:14.607534 I | ceph-object-zonegroup-controller: successfully started 2022-03-08T23:01:20.322 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:01:14.607560 I | ceph-object-zone-controller: successfully started 2022-03-08T23:01:20.323 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:01:14.607736 I | ceph-object-controller: successfully started 2022-03-08T23:01:20.323 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:01:14.607794 I | ceph-file-controller: successfully started 2022-03-08T23:01:20.323 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:01:14.607844 I | ceph-nfs-controller: successfully started 2022-03-08T23:01:20.323 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:01:14.607891 I | ceph-rbd-mirror-controller: successfully started 2022-03-08T23:01:20.323 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:01:14.607939 I | ceph-client-controller: successfully started 2022-03-08T23:01:20.324 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:01:14.607972 I | ceph-filesystem-mirror-controller: successfully started 2022-03-08T23:01:20.324 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:01:14.608021 I | operator: rook-ceph-operator-config-controller successfully started 2022-03-08T23:01:20.324 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:01:14.608055 I | ceph-csi: rook-ceph-operator-csi-controller successfully started 2022-03-08T23:01:20.324 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:01:14.608093 I | op-bucket-prov: rook-ceph-operator-bucket-controller successfully started 2022-03-08T23:01:20.324 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:01:14.608121 I | ceph-bucket-topic: successfully started 2022-03-08T23:01:20.325 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:01:14.608143 I | ceph-bucket-notification: successfully started 2022-03-08T23:01:20.325 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:01:14.608162 I | ceph-bucket-notification: successfully started 2022-03-08T23:01:20.325 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:01:14.608183 I | ceph-fs-subvolumegroup-controller: successfully started 2022-03-08T23:01:20.325 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:01:14.609658 I | operator: starting the controller-runtime manager 2022-03-08T23:01:20.325 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:01:14.713401 I | op-k8sutil: ROOK_CEPH_COMMANDS_TIMEOUT_SECONDS="15" (configmap) 2022-03-08T23:01:20.326 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:01:14.713437 I | op-k8sutil: ROOK_LOG_LEVEL="INFO" (configmap) 2022-03-08T23:01:20.326 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:01:14.713454 I | op-k8sutil: ROOK_ENABLE_DISCOVERY_DAEMON="false" (configmap) 2022-03-08T23:01:20.326 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:01:14.717649 I | operator: rook-ceph-operator-config-controller done reconciling 2022-03-08T23:01:20.326 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:01:14.720670 I | ceph-csi: CSI Ceph RBD driver disabled 2022-03-08T23:01:20.327 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:01:14.720692 I | op-k8sutil: removing daemonset csi-rbdplugin if it exists 2022-03-08T23:01:20.327 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:01:14.723294 I | op-k8sutil: removing deployment csi-rbdplugin-provisioner if it exists 2022-03-08T23:01:20.327 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:01:14.730055 I | ceph-csi: successfully removed CSI Ceph RBD driver 2022-03-08T23:01:20.327 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:01:14.730080 I | ceph-csi: CSI CephFS driver disabled 2022-03-08T23:01:20.327 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:01:14.730094 I | op-k8sutil: removing daemonset csi-cephfsplugin if it exists 2022-03-08T23:01:20.328 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:01:14.732043 I | op-k8sutil: removing deployment csi-cephfsplugin-provisioner if it exists 2022-03-08T23:01:20.328 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:01:14.738258 I | ceph-csi: successfully removed CSI CephFS driver 2022-03-08T23:01:21.171 INFO:teuthology.orchestra.run.smithi077.stdout:configmap/rook-config-override created 2022-03-08T23:01:21.176 INFO:tasks.rook:Cluster: apiVersion: ceph.rook.io/v1 kind: CephCluster metadata: name: rook-ceph namespace: rook-ceph spec: cephVersion: allowUnsupported: true image: quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3 dataDirHostPath: /var/lib/rook mgr: count: 1 modules: - enabled: true name: rook mon: allowMultiplePerNode: true count: 3 skipUpgradeChecks: true 2022-03-08T23:01:21.177 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2022-03-08T23:01:21.177 DEBUG:teuthology.orchestra.run.smithi077:> dd of=cluster.yaml 2022-03-08T23:01:21.183 DEBUG:teuthology.orchestra.run.smithi077:> kubectl create -f cluster.yaml 2022-03-08T23:01:23.289 INFO:teuthology.orchestra.run.smithi077.stdout:cephcluster.ceph.rook.io/rook-ceph created 2022-03-08T23:01:23.295 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:01:23.308684 I | ceph-spec: adding finalizer "cephcluster.ceph.rook.io" on "rook-ceph" 2022-03-08T23:01:23.300 DEBUG:teuthology.orchestra.remote:smithi077:rook/deploy/examples/operator.yaml is 20KB 2022-03-08T23:01:23.306 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:01:23.319574 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-03-08T23:01:23.311 DEBUG:teuthology.orchestra.run.smithi077:> kubectl create -f rook/deploy/examples/toolbox.yaml 2022-03-08T23:01:23.331 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:01:23.344306 I | ceph-cluster-controller: clusterInfo not yet found, must be a new cluster. 2022-03-08T23:01:23.344 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:01:23.357627 I | ceph-csi: successfully created csi config map "rook-ceph-csi-config" 2022-03-08T23:01:23.349 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:01:23.363329 I | ceph-spec: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3... 2022-03-08T23:01:23.440 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:01:23.452658 I | op-k8sutil: ROOK_CSI_ENABLE_RBD="true" (configmap) 2022-03-08T23:01:23.441 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:01:23.452683 I | op-k8sutil: ROOK_CSI_ENABLE_CEPHFS="true" (configmap) 2022-03-08T23:01:23.441 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:01:23.452692 I | op-k8sutil: ROOK_CSI_ALLOW_UNSUPPORTED_VERSION="false" (configmap) 2022-03-08T23:01:23.441 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:01:23.452700 I | op-k8sutil: ROOK_CSI_ENABLE_GRPC_METRICS="false" (configmap) 2022-03-08T23:01:23.442 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:01:23.452710 I | op-k8sutil: CSI_ENABLE_HOST_NETWORK="true" (default) 2022-03-08T23:01:23.442 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:01:23.452723 I | op-k8sutil: ROOK_CSI_CEPH_IMAGE="quay.io/cephcsi/cephcsi:v3.5.1" (default) 2022-03-08T23:01:23.442 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:01:23.452740 I | op-k8sutil: ROOK_CSI_REGISTRAR_IMAGE="k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.5.0" (default) 2022-03-08T23:01:23.442 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:01:23.452755 I | op-k8sutil: ROOK_CSI_PROVISIONER_IMAGE="k8s.gcr.io/sig-storage/csi-provisioner:v3.1.0" (default) 2022-03-08T23:01:23.443 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:01:23.452820 I | op-k8sutil: ROOK_CSI_ATTACHER_IMAGE="k8s.gcr.io/sig-storage/csi-attacher:v3.4.0" (default) 2022-03-08T23:01:23.443 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:01:23.452912 I | op-k8sutil: ROOK_CSI_SNAPSHOTTER_IMAGE="k8s.gcr.io/sig-storage/csi-snapshotter:v5.0.1" (default) 2022-03-08T23:01:23.443 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:01:23.452929 I | op-k8sutil: ROOK_CSI_KUBELET_DIR_PATH="/var/lib/kubelet" (default) 2022-03-08T23:01:23.443 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:01:23.452945 I | op-k8sutil: CSI_VOLUME_REPLICATION_IMAGE="quay.io/csiaddons/volumereplication-operator:v0.3.0" (default) 2022-03-08T23:01:23.444 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:01:23.452960 I | op-k8sutil: ROOK_CSIADDONS_IMAGE="quay.io/csiaddons/k8s-sidecar:v0.2.1" (default) 2022-03-08T23:01:23.444 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:01:23.452971 I | op-k8sutil: ROOK_CSI_CEPHFS_POD_LABELS="" (default) 2022-03-08T23:01:23.444 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:01:23.452984 I | op-k8sutil: ROOK_CSI_RBD_POD_LABELS="" (default) 2022-03-08T23:01:23.444 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:01:23.452997 I | ceph-csi: detecting the ceph csi image version for image "quay.io/cephcsi/cephcsi:v3.5.1" 2022-03-08T23:01:23.444 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:01:23.453120 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-03-08T23:01:23.445 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:01:23.453139 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-08T23:01:23.592 INFO:teuthology.orchestra.run.smithi077.stdout:deployment.apps/rook-ceph-tools created 2022-03-08T23:01:23.598 INFO:tasks.rook:Waiting for tools container to start 2022-03-08T23:01:23.598 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T23:01:23.734 INFO:teuthology.orchestra.run.smithi077.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T23:01:23.735 INFO:teuthology.orchestra.run.smithi077.stdout:rook-ceph-tools-d6d7c985c-vmw72 0/1 Pending 0 0s 2022-03-08T23:01:28.741 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T23:01:28.809 INFO:teuthology.orchestra.run.smithi077.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T23:01:28.809 INFO:teuthology.orchestra.run.smithi077.stdout:rook-ceph-tools-d6d7c985c-vmw72 0/1 ContainerCreating 0 5s 2022-03-08T23:01:33.812 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T23:01:33.882 INFO:teuthology.orchestra.run.smithi077.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T23:01:33.883 INFO:teuthology.orchestra.run.smithi077.stdout:rook-ceph-tools-d6d7c985c-vmw72 0/1 ContainerCreating 0 10s 2022-03-08T23:01:38.885 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T23:01:38.948 INFO:teuthology.orchestra.run.smithi077.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T23:01:38.949 INFO:teuthology.orchestra.run.smithi077.stdout:rook-ceph-tools-d6d7c985c-vmw72 0/1 ContainerCreating 0 15s 2022-03-08T23:01:43.951 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T23:01:44.012 INFO:teuthology.orchestra.run.smithi077.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T23:01:44.013 INFO:teuthology.orchestra.run.smithi077.stdout:rook-ceph-tools-d6d7c985c-vmw72 0/1 ContainerCreating 0 21s 2022-03-08T23:01:49.016 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T23:01:49.076 INFO:teuthology.orchestra.run.smithi077.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T23:01:49.076 INFO:teuthology.orchestra.run.smithi077.stdout:rook-ceph-tools-d6d7c985c-vmw72 0/1 ContainerCreating 0 26s 2022-03-08T23:01:54.080 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T23:01:54.139 INFO:teuthology.orchestra.run.smithi077.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T23:01:54.139 INFO:teuthology.orchestra.run.smithi077.stdout:rook-ceph-tools-d6d7c985c-vmw72 0/1 ContainerCreating 0 31s 2022-03-08T23:01:59.142 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T23:01:59.205 INFO:teuthology.orchestra.run.smithi077.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T23:01:59.205 INFO:teuthology.orchestra.run.smithi077.stdout:rook-ceph-tools-d6d7c985c-vmw72 0/1 ContainerCreating 0 36s 2022-03-08T23:02:04.206 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T23:02:04.273 INFO:teuthology.orchestra.run.smithi077.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T23:02:04.274 INFO:teuthology.orchestra.run.smithi077.stdout:rook-ceph-tools-d6d7c985c-vmw72 0/1 ContainerCreating 0 41s 2022-03-08T23:02:09.276 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T23:02:09.360 INFO:teuthology.orchestra.run.smithi077.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T23:02:09.360 INFO:teuthology.orchestra.run.smithi077.stdout:rook-ceph-tools-d6d7c985c-vmw72 0/1 ContainerCreating 0 46s 2022-03-08T23:02:14.362 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T23:02:14.430 INFO:teuthology.orchestra.run.smithi077.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T23:02:14.431 INFO:teuthology.orchestra.run.smithi077.stdout:rook-ceph-tools-d6d7c985c-vmw72 0/1 ContainerCreating 0 51s 2022-03-08T23:02:19.433 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T23:02:19.529 INFO:teuthology.orchestra.run.smithi077.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T23:02:19.529 INFO:teuthology.orchestra.run.smithi077.stdout:rook-ceph-tools-d6d7c985c-vmw72 0/1 ContainerCreating 0 56s 2022-03-08T23:02:24.532 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T23:02:24.594 INFO:teuthology.orchestra.run.smithi077.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T23:02:24.594 INFO:teuthology.orchestra.run.smithi077.stdout:rook-ceph-tools-d6d7c985c-vmw72 0/1 ContainerCreating 0 61s 2022-03-08T23:02:29.597 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T23:02:29.677 INFO:teuthology.orchestra.run.smithi077.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T23:02:29.678 INFO:teuthology.orchestra.run.smithi077.stdout:rook-ceph-tools-d6d7c985c-vmw72 0/1 ContainerCreating 0 66s 2022-03-08T23:02:34.681 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T23:02:34.749 INFO:teuthology.orchestra.run.smithi077.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T23:02:34.750 INFO:teuthology.orchestra.run.smithi077.stdout:rook-ceph-tools-d6d7c985c-vmw72 0/1 ContainerCreating 0 71s 2022-03-08T23:02:39.752 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T23:02:39.824 INFO:teuthology.orchestra.run.smithi077.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T23:02:39.825 INFO:teuthology.orchestra.run.smithi077.stdout:rook-ceph-tools-d6d7c985c-vmw72 0/1 ContainerCreating 0 76s 2022-03-08T23:02:44.828 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T23:02:44.889 INFO:teuthology.orchestra.run.smithi077.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T23:02:44.889 INFO:teuthology.orchestra.run.smithi077.stdout:rook-ceph-tools-d6d7c985c-vmw72 0/1 ContainerCreating 0 81s 2022-03-08T23:02:49.891 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T23:02:49.955 INFO:teuthology.orchestra.run.smithi077.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T23:02:49.955 INFO:teuthology.orchestra.run.smithi077.stdout:rook-ceph-tools-d6d7c985c-vmw72 0/1 ContainerCreating 0 86s 2022-03-08T23:02:54.958 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T23:02:55.024 INFO:teuthology.orchestra.run.smithi077.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T23:02:55.025 INFO:teuthology.orchestra.run.smithi077.stdout:rook-ceph-tools-d6d7c985c-vmw72 0/1 ContainerCreating 0 92s 2022-03-08T23:03:00.026 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T23:03:00.088 INFO:teuthology.orchestra.run.smithi077.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T23:03:00.088 INFO:teuthology.orchestra.run.smithi077.stdout:rook-ceph-tools-d6d7c985c-vmw72 0/1 ContainerCreating 0 97s 2022-03-08T23:03:05.090 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T23:03:05.152 INFO:teuthology.orchestra.run.smithi077.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T23:03:05.152 INFO:teuthology.orchestra.run.smithi077.stdout:rook-ceph-tools-d6d7c985c-vmw72 0/1 ContainerCreating 0 102s 2022-03-08T23:03:10.154 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T23:03:10.214 INFO:teuthology.orchestra.run.smithi077.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T23:03:10.215 INFO:teuthology.orchestra.run.smithi077.stdout:rook-ceph-tools-d6d7c985c-vmw72 0/1 ContainerCreating 0 107s 2022-03-08T23:03:12.150 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:03:12.147968 I | ceph-csi: Detected ceph CSI image version: "v3.5.1" 2022-03-08T23:03:12.150 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:03:12.148010 I | op-k8sutil: CSI_FORCE_CEPHFS_KERNEL_CLIENT="true" (configmap) 2022-03-08T23:03:12.150 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:03:12.148027 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default) 2022-03-08T23:03:12.151 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:03:12.148039 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default) 2022-03-08T23:03:12.151 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:03:12.148051 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default) 2022-03-08T23:03:12.151 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:03:12.148062 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default) 2022-03-08T23:03:12.151 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:03:12.148075 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default) 2022-03-08T23:03:12.152 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:03:12.148088 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default) 2022-03-08T23:03:12.152 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:03:12.148101 I | op-k8sutil: CSIADDONS_PORT="9070" (default) 2022-03-08T23:03:12.152 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:03:12.148115 I | op-k8sutil: CSIADDONS_PORT="9070" (default) 2022-03-08T23:03:12.152 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:03:12.148129 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default) 2022-03-08T23:03:12.152 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:03:12.148142 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default) 2022-03-08T23:03:12.153 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:03:12.148156 I | op-k8sutil: CSI_PLUGIN_PRIORITY_CLASSNAME="" (default) 2022-03-08T23:03:12.153 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:03:12.148169 I | op-k8sutil: CSI_PROVISIONER_PRIORITY_CLASSNAME="" (default) 2022-03-08T23:03:12.153 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:03:12.148182 I | op-k8sutil: CSI_ENABLE_OMAP_GENERATOR="false" (default) 2022-03-08T23:03:12.153 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:03:12.148195 I | op-k8sutil: CSI_ENABLE_RBD_SNAPSHOTTER="true" (configmap) 2022-03-08T23:03:12.153 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:03:12.148208 I | op-k8sutil: CSI_ENABLE_CEPHFS_SNAPSHOTTER="true" (configmap) 2022-03-08T23:03:12.154 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:03:12.148221 I | op-k8sutil: CSI_ENABLE_VOLUME_REPLICATION="false" (configmap) 2022-03-08T23:03:12.154 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:03:12.148235 I | op-k8sutil: CSI_ENABLE_CSIADDONS="false" (configmap) 2022-03-08T23:03:12.155 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:03:12.148249 I | op-k8sutil: CSI_CEPHFS_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-03-08T23:03:12.155 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:03:12.148265 I | op-k8sutil: CSI_RBD_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-03-08T23:03:12.155 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:03:12.148277 I | op-k8sutil: CSI_PLUGIN_ENABLE_SELINUX_HOST_MOUNT="false" (configmap) 2022-03-08T23:03:12.155 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:03:12.148289 I | ceph-csi: Kubernetes version is 1.23 2022-03-08T23:03:12.156 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:03:12.148305 I | op-k8sutil: ROOK_CSI_RESIZER_IMAGE="k8s.gcr.io/sig-storage/csi-resizer:v1.4.0" (default) 2022-03-08T23:03:12.156 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:03:12.148318 I | op-k8sutil: CSI_LOG_LEVEL="" (default) 2022-03-08T23:03:12.156 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:03:12.157432 I | op-k8sutil: CSI_PROVISIONER_REPLICAS="2" (configmap) 2022-03-08T23:03:12.171 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:03:12.169043 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-03-08T23:03:12.172 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:03:12.169068 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-08T23:03:12.172 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:03:12.169073 I | op-k8sutil: CSI_PLUGIN_TOLERATIONS="" (default) 2022-03-08T23:03:12.173 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:03:12.169078 I | op-k8sutil: CSI_PLUGIN_NODE_AFFINITY="" (default) 2022-03-08T23:03:12.173 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:03:12.169082 I | op-k8sutil: CSI_RBD_PLUGIN_TOLERATIONS="" (default) 2022-03-08T23:03:12.174 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:03:12.169087 I | op-k8sutil: CSI_RBD_PLUGIN_NODE_AFFINITY="" (default) 2022-03-08T23:03:12.174 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:03:12.169092 I | op-k8sutil: CSI_RBD_PLUGIN_RESOURCE="" (default) 2022-03-08T23:03:12.263 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:03:12.264165 I | op-k8sutil: CSI_RBD_PROVISIONER_TOLERATIONS="" (default) 2022-03-08T23:03:12.264 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:03:12.264197 I | op-k8sutil: CSI_RBD_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-08T23:03:12.264 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:03:12.264207 I | op-k8sutil: CSI_RBD_PROVISIONER_RESOURCE="" (default) 2022-03-08T23:03:12.316 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:03:12.316820 I | ceph-csi: successfully started CSI Ceph RBD driver 2022-03-08T23:03:12.420 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:03:12.420246 I | op-k8sutil: CSI_CEPHFS_PLUGIN_TOLERATIONS="" (default) 2022-03-08T23:03:12.420 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:03:12.420277 I | op-k8sutil: CSI_CEPHFS_PLUGIN_NODE_AFFINITY="" (default) 2022-03-08T23:03:12.421 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:03:12.420286 I | op-k8sutil: CSI_CEPHFS_PLUGIN_RESOURCE="" (default) 2022-03-08T23:03:12.459 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:03:12.459376 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_TOLERATIONS="" (default) 2022-03-08T23:03:12.459 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:03:12.459410 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-08T23:03:12.460 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:03:12.459420 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_RESOURCE="" (default) 2022-03-08T23:03:12.599 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:03:12.595802 I | ceph-csi: successfully started CSI CephFS driver 2022-03-08T23:03:12.764 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:03:12.762616 I | op-k8sutil: CSI_RBD_FSGROUPPOLICY="ReadWriteOnceWithFSType" (configmap) 2022-03-08T23:03:12.827 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:03:12.827977 I | ceph-csi: CSIDriver object created for driver "rook-ceph.rbd.csi.ceph.com" 2022-03-08T23:03:12.828 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:03:12.828006 I | op-k8sutil: CSI_CEPHFS_FSGROUPPOLICY="ReadWriteOnceWithFSType" (configmap) 2022-03-08T23:03:12.924 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:03:12.925375 I | ceph-csi: CSIDriver object created for driver "rook-ceph.cephfs.csi.ceph.com" 2022-03-08T23:03:15.217 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T23:03:15.306 INFO:teuthology.orchestra.run.smithi077.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T23:03:15.306 INFO:teuthology.orchestra.run.smithi077.stdout:rook-ceph-tools-d6d7c985c-vmw72 0/1 ContainerCreating 0 112s 2022-03-08T23:03:15.987 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:03:15.987140 I | ceph-spec: detected ceph image version: "17.0.0-11025 quincy" 2022-03-08T23:03:15.987 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:03:15.987165 I | ceph-cluster-controller: validating ceph version from provided image 2022-03-08T23:03:15.987 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:03:15.987176 W | ceph-cluster-controller: unsupported ceph version detected: "17.0.0-11025 quincy", pursuing 2022-03-08T23:03:16.011 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:03:16.012361 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-11025 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3" 2022-03-08T23:03:16.112 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:03:16.113260 E | ceph-spec: failed to update cluster condition to {Type:Progressing Status:True Reason:ClusterProgressing Message:Configuring the Ceph cluster LastHeartbeatTime:2022-03-08 23:03:16.042196553 +0000 UTC m=+121.858923011 LastTransitionTime:2022-03-08 23:03:16.042196366 +0000 UTC m=+121.858922848}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-08T23:03:16.152 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:03:16.153086 I | op-mon: start running mons 2022-03-08T23:03:16.271 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:03:16.271398 I | op-mon: creating mon secrets for a new cluster 2022-03-08T23:03:16.287 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:03:16.288360 I | op-mon: existing maxMonID not found or failed to load. configmaps "rook-ceph-mon-endpoints" not found 2022-03-08T23:03:16.291 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:03:16.290685 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":[]}] data: mapping:{"node":{}} maxMonId:-1] 2022-03-08T23:03:16.943 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:03:16.943730 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-08T23:03:16.943 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:03:16.943936 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-08T23:03:18.618 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:03:18.618826 I | op-mon: targeting the mon count 3 2022-03-08T23:03:18.685 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:03:18.686477 I | op-mon: created canary deployment rook-ceph-mon-a-canary 2022-03-08T23:03:18.691 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:03:18.692135 I | op-mon: created canary deployment rook-ceph-mon-b-canary 2022-03-08T23:03:18.726 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:03:18.727040 I | op-mon: created canary deployment rook-ceph-mon-c-canary 2022-03-08T23:03:19.347 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:03:19.347495 I | op-mon: canary monitor deployment rook-ceph-mon-b-canary scheduled to smithi106 2022-03-08T23:03:19.347 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:03:19.347530 I | op-mon: mon b assigned to node smithi106 2022-03-08T23:03:19.545 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:03:19.545892 I | op-mon: canary monitor deployment rook-ceph-mon-c-canary scheduled to smithi077 2022-03-08T23:03:19.618 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:03:19.545926 I | op-mon: mon c assigned to node smithi077 2022-03-08T23:03:20.309 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T23:03:20.658 INFO:teuthology.orchestra.run.smithi077.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T23:03:20.659 INFO:teuthology.orchestra.run.smithi077.stdout:rook-ceph-tools-d6d7c985c-vmw72 0/1 ContainerCreating 0 117s 2022-03-08T23:03:23.429 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:03:23.429618 I | op-mon: parsing mon endpoints: 2022-03-08T23:03:23.429 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:03:23.429648 W | op-mon: ignoring invalid monitor 2022-03-08T23:03:23.429 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:03:23.429679 I | op-k8sutil: ROOK_OBC_WATCH_OPERATOR_NAMESPACE="true" (configmap) 2022-03-08T23:03:23.430 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:03:23.429691 I | op-bucket-prov: ceph bucket provisioner launched watching for provisioner "rook-ceph.ceph.rook.io/bucket" 2022-03-08T23:03:23.434 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:03:23.432188 I | op-bucket-prov: successfully reconciled bucket provisioner 2022-03-08T23:03:23.435 INFO:tasks.rook.operator.smithi077.stdout:I0308 23:03:23.432306 1 manager.go:135] objectbucket.io/provisioner-manager "msg"="starting provisioner" "name"="rook-ceph.ceph.rook.io/bucket" 2022-03-08T23:03:23.800 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:03:23.801071 I | op-mon: canary monitor deployment rook-ceph-mon-a-canary scheduled to smithi135 2022-03-08T23:03:23.801 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:03:23.801103 I | op-mon: mon a assigned to node smithi135 2022-03-08T23:03:23.810 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:03:23.808507 I | op-mon: cleaning up canary monitor deployment "rook-ceph-mon-a-canary" 2022-03-08T23:03:23.813 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:03:23.813537 I | op-mon: cleaning up canary monitor deployment "rook-ceph-mon-b-canary" 2022-03-08T23:03:23.906 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:03:23.906748 I | op-mon: cleaning up canary monitor deployment "rook-ceph-mon-c-canary" 2022-03-08T23:03:23.954 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:03:23.955281 I | op-mon: creating mon a 2022-03-08T23:03:24.104 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:03:24.104989 I | op-mon: mon "a" endpoint is [v2:10.110.141.123:3300,v1:10.110.141.123:6789] 2022-03-08T23:03:24.295 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:03:24.296242 I | op-mon: monitor endpoints changed, updating the bootstrap peer token 2022-03-08T23:03:24.296 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:03:24.296367 I | op-mon: monitor endpoints changed, updating the bootstrap peer token 2022-03-08T23:03:24.300 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:03:24.296848 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.110.141.123:6789"]}] data:a=10.110.141.123:6789 mapping:{"node":{"a":{"Name":"smithi135","Hostname":"smithi135","Address":"172.21.15.135"},"b":{"Name":"smithi106","Hostname":"smithi106","Address":"172.21.15.106"},"c":{"Name":"smithi077","Hostname":"smithi077","Address":"172.21.15.77"}}} maxMonId:-1] 2022-03-08T23:03:24.403 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:03:24.404335 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-08T23:03:24.407 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:03:24.404606 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-08T23:03:24.465 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:03:24.466611 I | op-mon: 0 of 1 expected mons are ready. creating or updating deployments without checking quorum in attempt to achieve a healthy mon cluster 2022-03-08T23:03:24.573 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:03:24.572899 I | op-mon: updating maxMonID from -1 to 0 after committing mon "a" 2022-03-08T23:03:25.025 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:03:25.026180 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.110.141.123:6789"]}] data:a=10.110.141.123:6789 mapping:{"node":{"a":{"Name":"smithi135","Hostname":"smithi135","Address":"172.21.15.135"},"b":{"Name":"smithi106","Hostname":"smithi106","Address":"172.21.15.106"},"c":{"Name":"smithi077","Hostname":"smithi077","Address":"172.21.15.77"}}} maxMonId:0] 2022-03-08T23:03:25.025 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:03:25.026215 I | op-mon: waiting for mon quorum with [a] 2022-03-08T23:03:25.228 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:03:25.229335 I | op-mon: mon a is not yet running 2022-03-08T23:03:25.229 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:03:25.229364 I | op-mon: mons running: [] 2022-03-08T23:03:25.662 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T23:03:25.726 INFO:teuthology.orchestra.run.smithi077.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T23:03:25.726 INFO:teuthology.orchestra.run.smithi077.stdout:rook-ceph-tools-d6d7c985c-vmw72 0/1 ContainerCreating 0 2m2s 2022-03-08T23:03:30.728 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T23:03:30.795 INFO:teuthology.orchestra.run.smithi077.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T23:03:30.795 INFO:teuthology.orchestra.run.smithi077.stdout:rook-ceph-tools-d6d7c985c-vmw72 0/1 ContainerCreating 0 2m7s 2022-03-08T23:03:35.797 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T23:03:35.867 INFO:teuthology.orchestra.run.smithi077.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T23:03:35.868 INFO:teuthology.orchestra.run.smithi077.stdout:rook-ceph-tools-d6d7c985c-vmw72 0/1 ContainerCreating 0 2m12s 2022-03-08T23:03:40.871 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T23:03:40.939 INFO:teuthology.orchestra.run.smithi077.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T23:03:40.939 INFO:teuthology.orchestra.run.smithi077.stdout:rook-ceph-tools-d6d7c985c-vmw72 0/1 ContainerCreating 0 2m17s 2022-03-08T23:03:45.372 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:03:45.373920 I | op-mon: mons running: [a] 2022-03-08T23:03:45.942 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T23:03:46.009 INFO:teuthology.orchestra.run.smithi077.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T23:03:46.010 INFO:teuthology.orchestra.run.smithi077.stdout:rook-ceph-tools-d6d7c985c-vmw72 1/1 Running 0 2m23s 2022-03-08T23:03:46.017 INFO:tasks.rook:Waiting for mgr/rook orchestrator to be available 2022-03-08T23:03:46.017 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph orch status -f json 2022-03-08T23:03:55.516 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-03-08T23:03:55.517 INFO:teuthology.orchestra.run.smithi077.stderr:no valid command found; 10 closest matches: 2022-03-08T23:03:55.518 INFO:teuthology.orchestra.run.smithi077.stderr:pg stat 2022-03-08T23:03:55.518 INFO:teuthology.orchestra.run.smithi077.stderr:pg getmap 2022-03-08T23:03:55.518 INFO:teuthology.orchestra.run.smithi077.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-08T23:03:55.519 INFO:teuthology.orchestra.run.smithi077.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-08T23:03:55.519 INFO:teuthology.orchestra.run.smithi077.stderr:pg dump_pools_json 2022-03-08T23:03:55.520 INFO:teuthology.orchestra.run.smithi077.stderr:pg ls-by-pool [...] 2022-03-08T23:03:55.520 INFO:teuthology.orchestra.run.smithi077.stderr:pg ls-by-primary [] [...] 2022-03-08T23:03:55.521 INFO:teuthology.orchestra.run.smithi077.stderr:pg ls-by-osd [] [...] 2022-03-08T23:03:55.521 INFO:teuthology.orchestra.run.smithi077.stderr:pg ls [] [...] 2022-03-08T23:03:55.522 INFO:teuthology.orchestra.run.smithi077.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-08T23:03:55.522 INFO:teuthology.orchestra.run.smithi077.stderr:Error EINVAL: invalid command 2022-03-08T23:03:55.523 INFO:teuthology.orchestra.run.smithi077.stderr:command terminated with exit code 22 2022-03-08T23:03:55.524 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-08T23:03:55.673 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:03:55.675183 I | op-mon: Monitors in quorum: [a] 2022-03-08T23:03:55.674 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:03:55.675209 I | op-mon: mons created: 1 2022-03-08T23:03:55.915 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:03:55.916766 I | op-mon: waiting for mon quorum with [a] 2022-03-08T23:03:55.927 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:03:55.929706 I | op-mon: mons running: [a] 2022-03-08T23:03:56.186 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:03:56.187834 I | op-mon: Monitors in quorum: [a] 2022-03-08T23:03:56.186 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:03:56.187858 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-08T23:03:56.475 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:03:56.476902 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-08T23:03:56.475 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:03:56.476925 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-03-08T23:03:56.765 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:03:56.766422 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-03-08T23:03:56.765 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:03:56.766449 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-08T23:03:57.050 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:03:57.052067 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-08T23:03:57.050 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:03:57.052087 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-03-08T23:03:57.314 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:03:57.315699 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-03-08T23:03:57.314 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:03:57.315721 I | op-config: deleting "log file" option from the mon configuration database 2022-03-08T23:03:57.547 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:03:57.548702 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-03-08T23:03:57.547 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:03:57.548736 I | op-mon: creating mon b 2022-03-08T23:03:57.659 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:03:57.661195 I | op-mon: mon "a" endpoint is [v2:10.110.141.123:3300,v1:10.110.141.123:6789] 2022-03-08T23:03:57.694 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:03:57.696036 I | op-mon: mon "b" endpoint is [v2:10.98.41.230:3300,v1:10.98.41.230:6789] 2022-03-08T23:03:57.794 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:03:57.795405 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.110.141.123:6789","10.98.41.230:6789"]}] data:a=10.110.141.123:6789,b=10.98.41.230:6789 mapping:{"node":{"a":{"Name":"smithi135","Hostname":"smithi135","Address":"172.21.15.135"},"b":{"Name":"smithi106","Hostname":"smithi106","Address":"172.21.15.106"},"c":{"Name":"smithi077","Hostname":"smithi077","Address":"172.21.15.77"}}} maxMonId:0] 2022-03-08T23:03:57.818 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:03:57.820087 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-08T23:03:57.821 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:03:57.820367 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-08T23:03:58.561 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:03:58.562465 I | op-mon: deployment for mon rook-ceph-mon-a already exists. updating if needed 2022-03-08T23:03:58.572 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:03:58.574674 I | op-k8sutil: deployment "rook-ceph-mon-a" did not change, nothing to update 2022-03-08T23:03:58.573 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:03:58.574703 I | op-mon: waiting for mon quorum with [a b] 2022-03-08T23:03:58.585 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:03:58.587496 I | op-mon: mons running: [a b] 2022-03-08T23:03:58.830 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:03:58.831563 I | op-mon: Monitors in quorum: [a] 2022-03-08T23:03:59.369 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:03:59.371210 I | op-mon: updating maxMonID from 0 to 1 after committing mon "b" 2022-03-08T23:04:00.780 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:04:00.781394 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.110.141.123:6789","10.98.41.230:6789"]}] data:a=10.110.141.123:6789,b=10.98.41.230:6789 mapping:{"node":{"a":{"Name":"smithi135","Hostname":"smithi135","Address":"172.21.15.135"},"b":{"Name":"smithi106","Hostname":"smithi106","Address":"172.21.15.106"},"c":{"Name":"smithi077","Hostname":"smithi077","Address":"172.21.15.77"}}} maxMonId:1] 2022-03-08T23:04:00.780 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:04:00.781429 I | op-mon: waiting for mon quorum with [a b] 2022-03-08T23:04:00.839 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:04:00.841358 I | op-mon: mon b is not yet running 2022-03-08T23:04:00.840 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:04:00.841382 I | op-mon: mons running: [a] 2022-03-08T23:04:01.077 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:04:01.079186 I | op-mon: Monitors in quorum: [a] 2022-03-08T23:04:01.077 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:04:01.079214 I | op-mon: mons created: 2 2022-03-08T23:04:01.324 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:04:01.326263 I | op-mon: waiting for mon quorum with [a b] 2022-03-08T23:04:01.339 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:04:01.341102 I | op-mon: mon b is not yet running 2022-03-08T23:04:01.339 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:04:01.341135 I | op-mon: mons running: [a] 2022-03-08T23:04:05.527 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph orch status -f json 2022-03-08T23:04:06.093 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-03-08T23:04:06.093 INFO:teuthology.orchestra.run.smithi077.stderr:no valid command found; 10 closest matches: 2022-03-08T23:04:06.094 INFO:teuthology.orchestra.run.smithi077.stderr:pg stat 2022-03-08T23:04:06.094 INFO:teuthology.orchestra.run.smithi077.stderr:pg getmap 2022-03-08T23:04:06.094 INFO:teuthology.orchestra.run.smithi077.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-08T23:04:06.095 INFO:teuthology.orchestra.run.smithi077.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-08T23:04:06.095 INFO:teuthology.orchestra.run.smithi077.stderr:pg dump_pools_json 2022-03-08T23:04:06.095 INFO:teuthology.orchestra.run.smithi077.stderr:pg ls-by-pool [...] 2022-03-08T23:04:06.095 INFO:teuthology.orchestra.run.smithi077.stderr:pg ls-by-primary [] [...] 2022-03-08T23:04:06.095 INFO:teuthology.orchestra.run.smithi077.stderr:pg ls-by-osd [] [...] 2022-03-08T23:04:06.096 INFO:teuthology.orchestra.run.smithi077.stderr:pg ls [] [...] 2022-03-08T23:04:06.096 INFO:teuthology.orchestra.run.smithi077.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-08T23:04:06.096 INFO:teuthology.orchestra.run.smithi077.stderr:Error EINVAL: invalid command 2022-03-08T23:04:06.097 INFO:teuthology.orchestra.run.smithi077.stderr:command terminated with exit code 22 2022-03-08T23:04:06.098 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-08T23:04:06.545 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:04:06.547643 I | op-mon: mon b is not yet running 2022-03-08T23:04:06.545 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:04:06.547671 I | op-mon: mons running: [a] 2022-03-08T23:04:11.561 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:04:11.562938 I | op-mon: mon b is not yet running 2022-03-08T23:04:11.561 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:04:11.562965 I | op-mon: mons running: [a] 2022-03-08T23:04:16.099 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph orch status -f json 2022-03-08T23:04:16.611 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-03-08T23:04:16.611 INFO:teuthology.orchestra.run.smithi077.stderr:no valid command found; 10 closest matches: 2022-03-08T23:04:16.612 INFO:teuthology.orchestra.run.smithi077.stderr:pg stat 2022-03-08T23:04:16.612 INFO:teuthology.orchestra.run.smithi077.stderr:pg getmap 2022-03-08T23:04:16.612 INFO:teuthology.orchestra.run.smithi077.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-08T23:04:16.612 INFO:teuthology.orchestra.run.smithi077.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-08T23:04:16.612 INFO:teuthology.orchestra.run.smithi077.stderr:pg dump_pools_json 2022-03-08T23:04:16.613 INFO:teuthology.orchestra.run.smithi077.stderr:pg ls-by-pool [...] 2022-03-08T23:04:16.613 INFO:teuthology.orchestra.run.smithi077.stderr:pg ls-by-primary [] [...] 2022-03-08T23:04:16.613 INFO:teuthology.orchestra.run.smithi077.stderr:pg ls-by-osd [] [...] 2022-03-08T23:04:16.613 INFO:teuthology.orchestra.run.smithi077.stderr:pg ls [] [...] 2022-03-08T23:04:16.614 INFO:teuthology.orchestra.run.smithi077.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-08T23:04:16.614 INFO:teuthology.orchestra.run.smithi077.stderr:Error EINVAL: invalid command 2022-03-08T23:04:16.614 INFO:teuthology.orchestra.run.smithi077.stderr:command terminated with exit code 22 2022-03-08T23:04:16.616 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-08T23:04:17.483 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:04:17.485224 I | op-mon: mon b is not yet running 2022-03-08T23:04:17.484 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:04:17.485263 I | op-mon: mons running: [a] 2022-03-08T23:04:22.738 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:04:22.740253 I | op-mon: mon b is not yet running 2022-03-08T23:04:22.738 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:04:22.740277 I | op-mon: mons running: [a] 2022-03-08T23:04:26.618 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph orch status -f json 2022-03-08T23:04:27.299 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-03-08T23:04:27.300 INFO:teuthology.orchestra.run.smithi077.stderr:no valid command found; 10 closest matches: 2022-03-08T23:04:27.300 INFO:teuthology.orchestra.run.smithi077.stderr:pg stat 2022-03-08T23:04:27.300 INFO:teuthology.orchestra.run.smithi077.stderr:pg getmap 2022-03-08T23:04:27.300 INFO:teuthology.orchestra.run.smithi077.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-08T23:04:27.301 INFO:teuthology.orchestra.run.smithi077.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-08T23:04:27.301 INFO:teuthology.orchestra.run.smithi077.stderr:pg dump_pools_json 2022-03-08T23:04:27.301 INFO:teuthology.orchestra.run.smithi077.stderr:pg ls-by-pool [...] 2022-03-08T23:04:27.301 INFO:teuthology.orchestra.run.smithi077.stderr:pg ls-by-primary [] [...] 2022-03-08T23:04:27.301 INFO:teuthology.orchestra.run.smithi077.stderr:pg ls-by-osd [] [...] 2022-03-08T23:04:27.302 INFO:teuthology.orchestra.run.smithi077.stderr:pg ls [] [...] 2022-03-08T23:04:27.302 INFO:teuthology.orchestra.run.smithi077.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-08T23:04:27.302 INFO:teuthology.orchestra.run.smithi077.stderr:Error EINVAL: invalid command 2022-03-08T23:04:27.303 INFO:teuthology.orchestra.run.smithi077.stderr:command terminated with exit code 22 2022-03-08T23:04:27.304 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-08T23:04:28.847 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:04:28.849937 I | op-mon: mon b is not yet running 2022-03-08T23:04:28.847 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:04:28.849960 I | op-mon: mons running: [a] 2022-03-08T23:04:35.193 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:04:35.196303 I | op-mon: mon b is not yet running 2022-03-08T23:04:35.194 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:04:35.196339 I | op-mon: mons running: [a] 2022-03-08T23:04:37.305 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph orch status -f json 2022-03-08T23:04:37.972 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-03-08T23:04:37.973 INFO:teuthology.orchestra.run.smithi077.stderr:no valid command found; 10 closest matches: 2022-03-08T23:04:37.973 INFO:teuthology.orchestra.run.smithi077.stderr:pg stat 2022-03-08T23:04:37.973 INFO:teuthology.orchestra.run.smithi077.stderr:pg getmap 2022-03-08T23:04:37.973 INFO:teuthology.orchestra.run.smithi077.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-08T23:04:37.974 INFO:teuthology.orchestra.run.smithi077.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-08T23:04:37.974 INFO:teuthology.orchestra.run.smithi077.stderr:pg dump_pools_json 2022-03-08T23:04:37.974 INFO:teuthology.orchestra.run.smithi077.stderr:pg ls-by-pool [...] 2022-03-08T23:04:37.974 INFO:teuthology.orchestra.run.smithi077.stderr:pg ls-by-primary [] [...] 2022-03-08T23:04:37.975 INFO:teuthology.orchestra.run.smithi077.stderr:pg ls-by-osd [] [...] 2022-03-08T23:04:37.975 INFO:teuthology.orchestra.run.smithi077.stderr:pg ls [] [...] 2022-03-08T23:04:37.975 INFO:teuthology.orchestra.run.smithi077.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-08T23:04:37.975 INFO:teuthology.orchestra.run.smithi077.stderr:Error EINVAL: invalid command 2022-03-08T23:04:37.977 INFO:teuthology.orchestra.run.smithi077.stderr:command terminated with exit code 22 2022-03-08T23:04:37.978 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-08T23:04:40.228 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:04:40.230799 I | op-mon: mon b is not yet running 2022-03-08T23:04:40.228 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:04:40.230827 I | op-mon: mons running: [a] 2022-03-08T23:04:45.259 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:04:45.261964 I | op-mon: mon b is not yet running 2022-03-08T23:04:45.259 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:04:45.262002 I | op-mon: mons running: [a] 2022-03-08T23:04:47.981 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph orch status -f json 2022-03-08T23:04:48.806 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-03-08T23:04:48.807 INFO:teuthology.orchestra.run.smithi077.stderr:no valid command found; 10 closest matches: 2022-03-08T23:04:48.807 INFO:teuthology.orchestra.run.smithi077.stderr:pg stat 2022-03-08T23:04:48.807 INFO:teuthology.orchestra.run.smithi077.stderr:pg getmap 2022-03-08T23:04:48.807 INFO:teuthology.orchestra.run.smithi077.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-08T23:04:48.808 INFO:teuthology.orchestra.run.smithi077.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-08T23:04:48.808 INFO:teuthology.orchestra.run.smithi077.stderr:pg dump_pools_json 2022-03-08T23:04:48.808 INFO:teuthology.orchestra.run.smithi077.stderr:pg ls-by-pool [...] 2022-03-08T23:04:48.808 INFO:teuthology.orchestra.run.smithi077.stderr:pg ls-by-primary [] [...] 2022-03-08T23:04:48.808 INFO:teuthology.orchestra.run.smithi077.stderr:pg ls-by-osd [] [...] 2022-03-08T23:04:48.809 INFO:teuthology.orchestra.run.smithi077.stderr:pg ls [] [...] 2022-03-08T23:04:48.809 INFO:teuthology.orchestra.run.smithi077.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-08T23:04:48.809 INFO:teuthology.orchestra.run.smithi077.stderr:Error EINVAL: invalid command 2022-03-08T23:04:48.810 INFO:teuthology.orchestra.run.smithi077.stderr:command terminated with exit code 22 2022-03-08T23:04:48.812 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-08T23:04:50.291 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:04:50.294008 I | op-mon: mon b is not yet running 2022-03-08T23:04:50.291 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:04:50.294042 I | op-mon: mons running: [a] 2022-03-08T23:04:55.316 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:04:55.319517 I | op-mon: mon b is not yet running 2022-03-08T23:04:55.316 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:04:55.319546 I | op-mon: mons running: [a] 2022-03-08T23:04:58.813 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph orch status -f json 2022-03-08T23:04:59.326 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-03-08T23:04:59.327 INFO:teuthology.orchestra.run.smithi077.stderr:no valid command found; 10 closest matches: 2022-03-08T23:04:59.327 INFO:teuthology.orchestra.run.smithi077.stderr:pg stat 2022-03-08T23:04:59.328 INFO:teuthology.orchestra.run.smithi077.stderr:pg getmap 2022-03-08T23:04:59.328 INFO:teuthology.orchestra.run.smithi077.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-08T23:04:59.328 INFO:teuthology.orchestra.run.smithi077.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-08T23:04:59.328 INFO:teuthology.orchestra.run.smithi077.stderr:pg dump_pools_json 2022-03-08T23:04:59.328 INFO:teuthology.orchestra.run.smithi077.stderr:pg ls-by-pool [...] 2022-03-08T23:04:59.329 INFO:teuthology.orchestra.run.smithi077.stderr:pg ls-by-primary [] [...] 2022-03-08T23:04:59.329 INFO:teuthology.orchestra.run.smithi077.stderr:pg ls-by-osd [] [...] 2022-03-08T23:04:59.329 INFO:teuthology.orchestra.run.smithi077.stderr:pg ls [] [...] 2022-03-08T23:04:59.329 INFO:teuthology.orchestra.run.smithi077.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-08T23:04:59.329 INFO:teuthology.orchestra.run.smithi077.stderr:Error EINVAL: invalid command 2022-03-08T23:04:59.331 INFO:teuthology.orchestra.run.smithi077.stderr:command terminated with exit code 22 2022-03-08T23:04:59.333 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-08T23:05:00.347 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:05:00.350484 I | op-mon: mon b is not yet running 2022-03-08T23:05:00.347 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:05:00.350521 I | op-mon: mons running: [a] 2022-03-08T23:05:05.415 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:05:05.419203 I | op-mon: mon b is not yet running 2022-03-08T23:05:05.416 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:05:05.419226 I | op-mon: mons running: [a] 2022-03-08T23:05:09.335 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph orch status -f json 2022-03-08T23:05:10.432 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:05:10.435888 I | op-mon: mons running: [a b] 2022-03-08T23:05:12.496 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-03-08T23:05:12.497 INFO:teuthology.orchestra.run.smithi077.stderr:no valid command found; 10 closest matches: 2022-03-08T23:05:12.497 INFO:teuthology.orchestra.run.smithi077.stderr:pg stat 2022-03-08T23:05:12.497 INFO:teuthology.orchestra.run.smithi077.stderr:pg getmap 2022-03-08T23:05:12.497 INFO:teuthology.orchestra.run.smithi077.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-08T23:05:12.498 INFO:teuthology.orchestra.run.smithi077.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-08T23:05:12.498 INFO:teuthology.orchestra.run.smithi077.stderr:pg dump_pools_json 2022-03-08T23:05:12.498 INFO:teuthology.orchestra.run.smithi077.stderr:pg ls-by-pool [...] 2022-03-08T23:05:12.498 INFO:teuthology.orchestra.run.smithi077.stderr:pg ls-by-primary [] [...] 2022-03-08T23:05:12.499 INFO:teuthology.orchestra.run.smithi077.stderr:pg ls-by-osd [] [...] 2022-03-08T23:05:12.499 INFO:teuthology.orchestra.run.smithi077.stderr:pg ls [] [...] 2022-03-08T23:05:12.499 INFO:teuthology.orchestra.run.smithi077.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-08T23:05:12.499 INFO:teuthology.orchestra.run.smithi077.stderr:Error EINVAL: invalid command 2022-03-08T23:05:12.500 INFO:teuthology.orchestra.run.smithi077.stderr:command terminated with exit code 22 2022-03-08T23:05:12.503 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-08T23:05:13.691 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:05:13.694924 I | op-mon: Monitors in quorum: [a b] 2022-03-08T23:05:13.692 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:05:13.694964 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-08T23:05:13.922 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:05:13.926125 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-08T23:05:13.923 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:05:13.926161 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-03-08T23:05:14.145 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:05:14.148868 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-03-08T23:05:14.145 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:05:14.148908 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-08T23:05:14.358 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:05:14.361584 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-08T23:05:14.359 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:05:14.361623 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-03-08T23:05:14.597 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:05:14.600591 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-03-08T23:05:14.597 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:05:14.600625 I | op-config: deleting "log file" option from the mon configuration database 2022-03-08T23:05:14.812 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:05:14.816206 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-03-08T23:05:14.813 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:05:14.816240 I | op-mon: creating mon c 2022-03-08T23:05:14.937 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:05:14.940645 I | op-mon: mon "a" endpoint is [v2:10.110.141.123:3300,v1:10.110.141.123:6789] 2022-03-08T23:05:15.116 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:05:15.119563 I | op-mon: mon "b" endpoint is [v2:10.98.41.230:3300,v1:10.98.41.230:6789] 2022-03-08T23:05:15.168 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:05:15.172089 I | op-mon: mon "c" endpoint is [v2:10.109.47.142:3300,v1:10.109.47.142:6789] 2022-03-08T23:05:15.267 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:05:15.271211 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.98.41.230:6789","10.109.47.142:6789","10.110.141.123:6789"]}] data:a=10.110.141.123:6789,b=10.98.41.230:6789,c=10.109.47.142:6789 mapping:{"node":{"a":{"Name":"smithi135","Hostname":"smithi135","Address":"172.21.15.135"},"b":{"Name":"smithi106","Hostname":"smithi106","Address":"172.21.15.106"},"c":{"Name":"smithi077","Hostname":"smithi077","Address":"172.21.15.77"}}} maxMonId:1] 2022-03-08T23:05:15.817 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:05:15.820910 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-08T23:05:15.822 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:05:15.821222 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-08T23:05:16.247 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:05:16.250497 I | op-mon: 2 of 3 expected mon deployments exist. creating new deployment(s). 2022-03-08T23:05:16.255 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:05:16.258768 I | op-mon: deployment for mon rook-ceph-mon-a already exists. updating if needed 2022-03-08T23:05:16.265 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:05:16.269342 I | op-k8sutil: deployment "rook-ceph-mon-a" did not change, nothing to update 2022-03-08T23:05:16.281 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:05:16.284987 I | op-mon: deployment for mon rook-ceph-mon-b already exists. updating if needed 2022-03-08T23:05:16.291 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:05:16.295509 I | op-k8sutil: deployment "rook-ceph-mon-b" did not change, nothing to update 2022-03-08T23:05:16.437 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:05:16.440321 I | op-mon: updating maxMonID from 1 to 2 after committing mon "c" 2022-03-08T23:05:17.240 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:05:17.243464 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.110.141.123:6789","10.98.41.230:6789","10.109.47.142:6789"]}] data:a=10.110.141.123:6789,b=10.98.41.230:6789,c=10.109.47.142:6789 mapping:{"node":{"a":{"Name":"smithi135","Hostname":"smithi135","Address":"172.21.15.135"},"b":{"Name":"smithi106","Hostname":"smithi106","Address":"172.21.15.106"},"c":{"Name":"smithi077","Hostname":"smithi077","Address":"172.21.15.77"}}} maxMonId:2] 2022-03-08T23:05:17.240 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:05:17.243507 I | op-mon: waiting for mon quorum with [a b c] 2022-03-08T23:05:17.858 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:05:17.861587 I | op-mon: mon c is not yet running 2022-03-08T23:05:17.858 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:05:17.861629 I | op-mon: mons running: [a b] 2022-03-08T23:05:18.100 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:05:18.104539 I | op-mon: Monitors in quorum: [a b] 2022-03-08T23:05:18.101 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:05:18.104568 I | op-mon: mons created: 3 2022-03-08T23:05:18.360 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:05:18.363298 I | op-mon: waiting for mon quorum with [a b c] 2022-03-08T23:05:18.419 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:05:18.422626 I | op-mon: mon c is not yet running 2022-03-08T23:05:18.419 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:05:18.422658 I | op-mon: mons running: [a b] 2022-03-08T23:05:22.505 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph orch status -f json 2022-03-08T23:05:22.981 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-03-08T23:05:22.982 INFO:teuthology.orchestra.run.smithi077.stderr:no valid command found; 10 closest matches: 2022-03-08T23:05:22.982 INFO:teuthology.orchestra.run.smithi077.stderr:pg stat 2022-03-08T23:05:22.982 INFO:teuthology.orchestra.run.smithi077.stderr:pg getmap 2022-03-08T23:05:22.982 INFO:teuthology.orchestra.run.smithi077.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-08T23:05:22.983 INFO:teuthology.orchestra.run.smithi077.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-08T23:05:22.983 INFO:teuthology.orchestra.run.smithi077.stderr:pg dump_pools_json 2022-03-08T23:05:22.983 INFO:teuthology.orchestra.run.smithi077.stderr:pg ls-by-pool [...] 2022-03-08T23:05:22.983 INFO:teuthology.orchestra.run.smithi077.stderr:pg ls-by-primary [] [...] 2022-03-08T23:05:22.984 INFO:teuthology.orchestra.run.smithi077.stderr:pg ls-by-osd [] [...] 2022-03-08T23:05:22.984 INFO:teuthology.orchestra.run.smithi077.stderr:pg ls [] [...] 2022-03-08T23:05:22.984 INFO:teuthology.orchestra.run.smithi077.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-08T23:05:22.984 INFO:teuthology.orchestra.run.smithi077.stderr:Error EINVAL: invalid command 2022-03-08T23:05:22.985 INFO:teuthology.orchestra.run.smithi077.stderr:command terminated with exit code 22 2022-03-08T23:05:22.986 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-08T23:05:23.439 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:05:23.442962 I | op-mon: mon c is not yet running 2022-03-08T23:05:23.439 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:05:23.442997 I | op-mon: mons running: [a b] 2022-03-08T23:05:28.465 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:05:28.469081 I | op-mon: mon c is not yet running 2022-03-08T23:05:28.465 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:05:28.469116 I | op-mon: mons running: [a b] 2022-03-08T23:05:32.987 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph orch status -f json 2022-03-08T23:05:33.475 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-03-08T23:05:33.475 INFO:teuthology.orchestra.run.smithi077.stderr:no valid command found; 10 closest matches: 2022-03-08T23:05:33.476 INFO:teuthology.orchestra.run.smithi077.stderr:pg stat 2022-03-08T23:05:33.476 INFO:teuthology.orchestra.run.smithi077.stderr:pg getmap 2022-03-08T23:05:33.476 INFO:teuthology.orchestra.run.smithi077.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-08T23:05:33.476 INFO:teuthology.orchestra.run.smithi077.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-08T23:05:33.476 INFO:teuthology.orchestra.run.smithi077.stderr:pg dump_pools_json 2022-03-08T23:05:33.477 INFO:teuthology.orchestra.run.smithi077.stderr:pg ls-by-pool [...] 2022-03-08T23:05:33.477 INFO:teuthology.orchestra.run.smithi077.stderr:pg ls-by-primary [] [...] 2022-03-08T23:05:33.477 INFO:teuthology.orchestra.run.smithi077.stderr:pg ls-by-osd [] [...] 2022-03-08T23:05:33.477 INFO:teuthology.orchestra.run.smithi077.stderr:pg ls [] [...] 2022-03-08T23:05:33.477 INFO:teuthology.orchestra.run.smithi077.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-08T23:05:33.478 INFO:teuthology.orchestra.run.smithi077.stderr:Error EINVAL: invalid command 2022-03-08T23:05:33.479 INFO:teuthology.orchestra.run.smithi077.stderr:command terminated with exit code 22 2022-03-08T23:05:33.481 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:05:33.485155 I | op-mon: mon c is not yet running 2022-03-08T23:05:33.481 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:05:33.485183 I | op-mon: mons running: [a b] 2022-03-08T23:05:33.482 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-08T23:05:38.654 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:05:38.658753 I | op-mon: mon c is not yet running 2022-03-08T23:05:38.654 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:05:38.658781 I | op-mon: mons running: [a b] 2022-03-08T23:05:43.483 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph orch status -f json 2022-03-08T23:05:43.956 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-03-08T23:05:43.957 INFO:teuthology.orchestra.run.smithi077.stderr:no valid command found; 10 closest matches: 2022-03-08T23:05:43.957 INFO:teuthology.orchestra.run.smithi077.stderr:pg stat 2022-03-08T23:05:43.957 INFO:teuthology.orchestra.run.smithi077.stderr:pg getmap 2022-03-08T23:05:43.957 INFO:teuthology.orchestra.run.smithi077.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-08T23:05:43.958 INFO:teuthology.orchestra.run.smithi077.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-08T23:05:43.958 INFO:teuthology.orchestra.run.smithi077.stderr:pg dump_pools_json 2022-03-08T23:05:43.958 INFO:teuthology.orchestra.run.smithi077.stderr:pg ls-by-pool [...] 2022-03-08T23:05:43.958 INFO:teuthology.orchestra.run.smithi077.stderr:pg ls-by-primary [] [...] 2022-03-08T23:05:43.959 INFO:teuthology.orchestra.run.smithi077.stderr:pg ls-by-osd [] [...] 2022-03-08T23:05:43.959 INFO:teuthology.orchestra.run.smithi077.stderr:pg ls [] [...] 2022-03-08T23:05:43.959 INFO:teuthology.orchestra.run.smithi077.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-08T23:05:43.959 INFO:teuthology.orchestra.run.smithi077.stderr:Error EINVAL: invalid command 2022-03-08T23:05:43.960 INFO:teuthology.orchestra.run.smithi077.stderr:command terminated with exit code 22 2022-03-08T23:05:43.962 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-08T23:05:44.708 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:05:44.712485 I | op-mon: mon c is not yet running 2022-03-08T23:05:44.708 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:05:44.712523 I | op-mon: mons running: [a b] 2022-03-08T23:05:50.174 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:05:50.178524 I | op-mon: mon c is not yet running 2022-03-08T23:05:50.174 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:05:50.178552 I | op-mon: mons running: [a b] 2022-03-08T23:05:53.964 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph orch status -f json 2022-03-08T23:05:54.478 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-03-08T23:05:54.478 INFO:teuthology.orchestra.run.smithi077.stderr:no valid command found; 10 closest matches: 2022-03-08T23:05:54.479 INFO:teuthology.orchestra.run.smithi077.stderr:pg stat 2022-03-08T23:05:54.479 INFO:teuthology.orchestra.run.smithi077.stderr:pg getmap 2022-03-08T23:05:54.479 INFO:teuthology.orchestra.run.smithi077.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-08T23:05:54.479 INFO:teuthology.orchestra.run.smithi077.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-08T23:05:54.480 INFO:teuthology.orchestra.run.smithi077.stderr:pg dump_pools_json 2022-03-08T23:05:54.480 INFO:teuthology.orchestra.run.smithi077.stderr:pg ls-by-pool [...] 2022-03-08T23:05:54.480 INFO:teuthology.orchestra.run.smithi077.stderr:pg ls-by-primary [] [...] 2022-03-08T23:05:54.480 INFO:teuthology.orchestra.run.smithi077.stderr:pg ls-by-osd [] [...] 2022-03-08T23:05:54.481 INFO:teuthology.orchestra.run.smithi077.stderr:pg ls [] [...] 2022-03-08T23:05:54.481 INFO:teuthology.orchestra.run.smithi077.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-08T23:05:54.481 INFO:teuthology.orchestra.run.smithi077.stderr:Error EINVAL: invalid command 2022-03-08T23:05:54.481 INFO:teuthology.orchestra.run.smithi077.stderr:command terminated with exit code 22 2022-03-08T23:05:54.483 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-08T23:05:55.196 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:05:55.200678 I | op-mon: mons running: [a b c] 2022-03-08T23:05:59.904 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:05:59.909040 I | op-mon: Monitors in quorum: [a b c] 2022-03-08T23:05:59.905 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:05:59.909082 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-08T23:06:00.145 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:00.149949 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-08T23:06:00.146 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:00.149978 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-03-08T23:06:00.381 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:00.386127 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-03-08T23:06:00.382 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:00.386165 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-08T23:06:00.614 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:00.618766 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-08T23:06:00.614 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:00.618791 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-03-08T23:06:00.840 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:00.844476 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-03-08T23:06:00.840 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:00.844514 I | op-config: deleting "log file" option from the mon configuration database 2022-03-08T23:06:01.096 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:01.101215 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-03-08T23:06:01.097 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:01.101273 I | cephclient: getting or creating ceph auth key "client.csi-rbd-provisioner" 2022-03-08T23:06:01.432 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:01.436750 I | cephclient: getting or creating ceph auth key "client.csi-rbd-node" 2022-03-08T23:06:01.782 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:01.786757 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-provisioner" 2022-03-08T23:06:02.259 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:02.263805 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-node" 2022-03-08T23:06:02.663 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:02.668100 I | ceph-csi: created kubernetes csi secrets for cluster "rook-ceph" 2022-03-08T23:06:02.663 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:02.668133 I | cephclient: getting or creating ceph auth key "client.crash" 2022-03-08T23:06:03.045 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:03.049844 I | ceph-crashcollector-controller: created kubernetes crash collector secret for cluster "rook-ceph" 2022-03-08T23:06:03.343 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:03.347344 I | cephclient: successfully enabled msgr2 protocol 2022-03-08T23:06:03.343 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:03.347416 I | cephclient: create rbd-mirror bootstrap peer token "client.rbd-mirror-peer" 2022-03-08T23:06:03.343 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:03.347427 I | cephclient: getting or creating ceph auth key "client.rbd-mirror-peer" 2022-03-08T23:06:03.669 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:03.673679 I | cephclient: successfully created rbd-mirror bootstrap peer token for cluster "rook-ceph" 2022-03-08T23:06:03.681 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:03.686142 I | op-mgr: start running mgr 2022-03-08T23:06:03.681 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:03.686172 I | cephclient: getting or creating ceph auth key "mgr.a" 2022-03-08T23:06:04.486 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph orch status -f json 2022-03-08T23:06:05.008 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-03-08T23:06:05.009 INFO:teuthology.orchestra.run.smithi077.stderr:no valid command found; 10 closest matches: 2022-03-08T23:06:05.009 INFO:teuthology.orchestra.run.smithi077.stderr:pg stat 2022-03-08T23:06:05.009 INFO:teuthology.orchestra.run.smithi077.stderr:pg getmap 2022-03-08T23:06:05.010 INFO:teuthology.orchestra.run.smithi077.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-08T23:06:05.010 INFO:teuthology.orchestra.run.smithi077.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-08T23:06:05.010 INFO:teuthology.orchestra.run.smithi077.stderr:pg dump_pools_json 2022-03-08T23:06:05.010 INFO:teuthology.orchestra.run.smithi077.stderr:pg ls-by-pool [...] 2022-03-08T23:06:05.011 INFO:teuthology.orchestra.run.smithi077.stderr:pg ls-by-primary [] [...] 2022-03-08T23:06:05.011 INFO:teuthology.orchestra.run.smithi077.stderr:pg ls-by-osd [] [...] 2022-03-08T23:06:05.011 INFO:teuthology.orchestra.run.smithi077.stderr:pg ls [] [...] 2022-03-08T23:06:05.011 INFO:teuthology.orchestra.run.smithi077.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-08T23:06:05.012 INFO:teuthology.orchestra.run.smithi077.stderr:Error EINVAL: invalid command 2022-03-08T23:06:05.012 INFO:teuthology.orchestra.run.smithi077.stderr:command terminated with exit code 22 2022-03-08T23:06:05.014 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-08T23:06:15.016 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph orch status -f json 2022-03-08T23:06:15.459 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-03-08T23:06:15.460 INFO:teuthology.orchestra.run.smithi077.stderr:no valid command found; 10 closest matches: 2022-03-08T23:06:15.460 INFO:teuthology.orchestra.run.smithi077.stderr:pg stat 2022-03-08T23:06:15.460 INFO:teuthology.orchestra.run.smithi077.stderr:pg getmap 2022-03-08T23:06:15.460 INFO:teuthology.orchestra.run.smithi077.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-08T23:06:15.461 INFO:teuthology.orchestra.run.smithi077.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-08T23:06:15.461 INFO:teuthology.orchestra.run.smithi077.stderr:pg dump_pools_json 2022-03-08T23:06:15.461 INFO:teuthology.orchestra.run.smithi077.stderr:pg ls-by-pool [...] 2022-03-08T23:06:15.461 INFO:teuthology.orchestra.run.smithi077.stderr:pg ls-by-primary [] [...] 2022-03-08T23:06:15.462 INFO:teuthology.orchestra.run.smithi077.stderr:pg ls-by-osd [] [...] 2022-03-08T23:06:15.462 INFO:teuthology.orchestra.run.smithi077.stderr:pg ls [] [...] 2022-03-08T23:06:15.462 INFO:teuthology.orchestra.run.smithi077.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-08T23:06:15.462 INFO:teuthology.orchestra.run.smithi077.stderr:Error EINVAL: invalid command 2022-03-08T23:06:15.464 INFO:teuthology.orchestra.run.smithi077.stderr:command terminated with exit code 22 2022-03-08T23:06:15.466 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-08T23:06:25.162 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:25.167952 I | op-k8sutil: finished waiting for updated deployment "rook-ceph-mgr-a" 2022-03-08T23:06:25.167 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:25.171809 I | op-mgr: setting services to point to mgr "a" 2022-03-08T23:06:25.253 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:25.257538 I | op-mgr: no need to update service "rook-ceph-mgr" 2022-03-08T23:06:25.257 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:25.257640 I | op-mgr: successful modules: balancer 2022-03-08T23:06:25.283 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:25.288321 I | op-osd: start running osds in namespace "rook-ceph" 2022-03-08T23:06:25.283 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:25.288353 W | op-osd: useAllNodes is set to false and no nodes, storageClassDevicesets or volumeSources are specified, no OSD pods are going to be created 2022-03-08T23:06:25.283 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:25.288365 I | op-osd: wait timeout for healthy OSDs during upgrade or restart is "10m0s" 2022-03-08T23:06:25.323 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:25.329277 I | op-osd: start provisioning the OSDs on PVCs, if needed 2022-03-08T23:06:25.334 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:25.339071 I | op-osd: no storageClassDeviceSets defined to configure OSDs on PVCs 2022-03-08T23:06:25.334 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:25.339113 I | op-osd: start provisioning the OSDs on nodes, if needed 2022-03-08T23:06:25.335 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:25.339120 I | op-osd: no nodes are defined for configuring OSDs on raw devices 2022-03-08T23:06:25.469 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph orch status -f json 2022-03-08T23:06:25.747 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:25.752396 I | op-osd: finished running OSDs in namespace "rook-ceph" 2022-03-08T23:06:25.747 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:25.752421 I | ceph-cluster-controller: done reconciling ceph cluster in namespace "rook-ceph" 2022-03-08T23:06:25.754 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:25.759409 I | ceph-cluster-controller: enabling ceph mon monitoring goroutine for cluster "rook-ceph" 2022-03-08T23:06:25.754 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:25.759452 I | ceph-cluster-controller: enabling ceph osd monitoring goroutine for cluster "rook-ceph" 2022-03-08T23:06:25.754 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:25.759470 I | ceph-cluster-controller: enabling ceph status monitoring goroutine for cluster "rook-ceph" 2022-03-08T23:06:25.998 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-03-08T23:06:25.999 INFO:teuthology.orchestra.run.smithi077.stderr:Error ENOENT: No orchestrator configured (try `ceph orch set backend`) 2022-03-08T23:06:26.004 INFO:teuthology.orchestra.run.smithi077.stderr:command terminated with exit code 2 2022-03-08T23:06:26.007 DEBUG:teuthology.orchestra.run:got remote process result: 2 2022-03-08T23:06:26.366 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:26.371581 I | op-mgr: successful modules: dashboard 2022-03-08T23:06:26.372 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:26.377899 I | op-mgr: successful modules: prometheus 2022-03-08T23:06:26.375 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:26.378578 I | op-mgr: successful modules: mgr module(s) from the spec 2022-03-08T23:06:34.900 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:34.905560 I | op-mgr: successful modules: orchestrator modules 2022-03-08T23:06:36.008 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph orch status -f json 2022-03-08T23:06:36.622 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-03-08T23:06:36.622 INFO:teuthology.orchestra.run.smithi077.stdout:{"available": true, "backend": "rook"} 2022-03-08T23:06:36.643 INFO:tasks.rook: mgr/rook orchestrator is active 2022-03-08T23:06:36.644 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph config set mgr mgr/rook/storage_class scratch 2022-03-08T23:06:37.310 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph orch apply osd --all-available-devices 2022-03-08T23:06:38.033 INFO:teuthology.orchestra.run.smithi077.stdout:['Success'] 2022-03-08T23:06:38.035 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:38.040269 I | ceph-cluster-controller: CR has changed for "rook-ceph". diff=  v1.ClusterSpec{ 2022-03-08T23:06:38.035 INFO:tasks.rook.operator.smithi077.stdout:   CephVersion: {Image: "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", AllowUnsupported: true}, 2022-03-08T23:06:38.035 INFO:tasks.rook.operator.smithi077.stdout:   Storage: v1.StorageScopeSpec{ 2022-03-08T23:06:38.035 INFO:tasks.rook.operator.smithi077.stdout:   ... // 3 identical fields 2022-03-08T23:06:38.036 INFO:tasks.rook.operator.smithi077.stdout:   Config: nil, 2022-03-08T23:06:38.036 INFO:tasks.rook.operator.smithi077.stdout:   Selection: {}, 2022-03-08T23:06:38.036 INFO:tasks.rook.operator.smithi077.stdout:-  StorageClassDeviceSets: nil, 2022-03-08T23:06:38.036 INFO:tasks.rook.operator.smithi077.stdout:+  StorageClassDeviceSets: []v1.StorageClassDeviceSet{ 2022-03-08T23:06:38.036 INFO:tasks.rook.operator.smithi077.stdout:+  { 2022-03-08T23:06:38.037 INFO:tasks.rook.operator.smithi077.stdout:+  Name: "smithi077-lv-1", 2022-03-08T23:06:38.037 INFO:tasks.rook.operator.smithi077.stdout:+  Count: 1, 2022-03-08T23:06:38.038 INFO:tasks.rook.operator.smithi077.stdout:+  VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-08T23:06:38.038 INFO:tasks.rook.operator.smithi077.stdout:+  }, 2022-03-08T23:06:38.039 INFO:tasks.rook.operator.smithi077.stdout:+  { 2022-03-08T23:06:38.039 INFO:tasks.rook.operator.smithi077.stdout:+  Name: "smithi077-lv-2", 2022-03-08T23:06:38.039 INFO:tasks.rook.operator.smithi077.stdout:+  Count: 1, 2022-03-08T23:06:38.039 INFO:tasks.rook.operator.smithi077.stdout:+  VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-08T23:06:38.040 INFO:tasks.rook.operator.smithi077.stdout:+  }, 2022-03-08T23:06:38.040 INFO:tasks.rook.operator.smithi077.stdout:+  { 2022-03-08T23:06:38.040 INFO:tasks.rook.operator.smithi077.stdout:+  Name: "smithi077-lv-3", 2022-03-08T23:06:38.040 INFO:tasks.rook.operator.smithi077.stdout:+  Count: 1, 2022-03-08T23:06:38.040 INFO:tasks.rook.operator.smithi077.stdout:+  VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-08T23:06:38.041 INFO:tasks.rook.operator.smithi077.stdout:+  }, 2022-03-08T23:06:38.046 INFO:tasks.rook.operator.smithi077.stdout:+  { 2022-03-08T23:06:38.046 INFO:tasks.rook.operator.smithi077.stdout:+  Name: "smithi077-lv-4", 2022-03-08T23:06:38.046 INFO:tasks.rook.operator.smithi077.stdout:+  Count: 1, 2022-03-08T23:06:38.047 INFO:tasks.rook.operator.smithi077.stdout:+  VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-08T23:06:38.047 INFO:tasks.rook.operator.smithi077.stdout:+  }, 2022-03-08T23:06:38.047 INFO:tasks.rook.operator.smithi077.stdout:+  { 2022-03-08T23:06:38.047 INFO:tasks.rook.operator.smithi077.stdout:+  Name: "smithi106-lv-1", 2022-03-08T23:06:38.047 INFO:tasks.rook.operator.smithi077.stdout:+  Count: 1, 2022-03-08T23:06:38.048 INFO:tasks.rook.operator.smithi077.stdout:+  VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-08T23:06:38.048 INFO:tasks.rook.operator.smithi077.stdout:+  }, 2022-03-08T23:06:38.048 INFO:tasks.rook.operator.smithi077.stdout:+  { 2022-03-08T23:06:38.048 INFO:tasks.rook.operator.smithi077.stdout:+  Name: "smithi106-lv-2", 2022-03-08T23:06:38.048 INFO:tasks.rook.operator.smithi077.stdout:+  Count: 1, 2022-03-08T23:06:38.049 INFO:tasks.rook.operator.smithi077.stdout:+  VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-08T23:06:38.049 INFO:tasks.rook.operator.smithi077.stdout:+  }, 2022-03-08T23:06:38.049 INFO:tasks.rook.operator.smithi077.stdout:+  { 2022-03-08T23:06:38.049 INFO:tasks.rook.operator.smithi077.stdout:+  Name: "smithi106-lv-3", 2022-03-08T23:06:38.050 INFO:tasks.rook.operator.smithi077.stdout:+  Count: 1, 2022-03-08T23:06:38.050 INFO:tasks.rook.operator.smithi077.stdout:+  VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-08T23:06:38.050 INFO:tasks.rook.operator.smithi077.stdout:+  }, 2022-03-08T23:06:38.050 INFO:tasks.rook.operator.smithi077.stdout:+  { 2022-03-08T23:06:38.050 INFO:tasks.rook.operator.smithi077.stdout:+  Name: "smithi106-lv-4", 2022-03-08T23:06:38.051 INFO:tasks.rook.operator.smithi077.stdout:+  Count: 1, 2022-03-08T23:06:38.051 INFO:tasks.rook.operator.smithi077.stdout:+  VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-08T23:06:38.051 INFO:tasks.rook.operator.smithi077.stdout:+  }, 2022-03-08T23:06:38.051 INFO:tasks.rook.operator.smithi077.stdout:+  { 2022-03-08T23:06:38.051 INFO:tasks.rook.operator.smithi077.stdout:+  Name: "smithi135-lv-1", 2022-03-08T23:06:38.052 INFO:tasks.rook.operator.smithi077.stdout:+  Count: 1, 2022-03-08T23:06:38.052 INFO:tasks.rook.operator.smithi077.stdout:+  VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-08T23:06:38.052 INFO:tasks.rook.operator.smithi077.stdout:+  }, 2022-03-08T23:06:38.052 INFO:tasks.rook.operator.smithi077.stdout:+  { 2022-03-08T23:06:38.053 INFO:tasks.rook.operator.smithi077.stdout:+  Name: "smithi135-lv-2", 2022-03-08T23:06:38.053 INFO:tasks.rook.operator.smithi077.stdout:+  Count: 1, 2022-03-08T23:06:38.053 INFO:tasks.rook.operator.smithi077.stdout:+  VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-08T23:06:38.053 INFO:tasks.rook.operator.smithi077.stdout:+  }, 2022-03-08T23:06:38.054 INFO:tasks.rook.operator.smithi077.stdout:+  { 2022-03-08T23:06:38.054 INFO:tasks.rook.operator.smithi077.stdout:+  Name: "smithi135-lv-3", 2022-03-08T23:06:38.054 INFO:tasks.rook.operator.smithi077.stdout:+  Count: 1, 2022-03-08T23:06:38.055 INFO:tasks.rook.operator.smithi077.stdout:+  VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-08T23:06:38.055 INFO:tasks.rook.operator.smithi077.stdout:+  }, 2022-03-08T23:06:38.055 INFO:tasks.rook.operator.smithi077.stdout:+  { 2022-03-08T23:06:38.055 INFO:tasks.rook.operator.smithi077.stdout:+  Name: "smithi135-lv-4", 2022-03-08T23:06:38.056 INFO:tasks.rook.operator.smithi077.stdout:+  Count: 1, 2022-03-08T23:06:38.056 INFO:tasks.rook.operator.smithi077.stdout:+  VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-08T23:06:38.056 INFO:tasks.rook.operator.smithi077.stdout:+  }, 2022-03-08T23:06:38.056 INFO:tasks.rook.operator.smithi077.stdout:+  }, 2022-03-08T23:06:38.057 INFO:tasks.rook.operator.smithi077.stdout:   }, 2022-03-08T23:06:38.057 INFO:tasks.rook.operator.smithi077.stdout:   Annotations: nil, 2022-03-08T23:06:38.057 INFO:tasks.rook.operator.smithi077.stdout:   Labels: nil, 2022-03-08T23:06:38.057 INFO:tasks.rook.operator.smithi077.stdout:   ... // 20 identical fields 2022-03-08T23:06:38.058 INFO:tasks.rook.operator.smithi077.stdout:  } 2022-03-08T23:06:38.058 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:38.040415 I | operator: reloading operator's CRDs manager, cancelling all orchestrations! 2022-03-08T23:06:38.058 INFO:tasks.rook.operator.smithi077.stdout:I0308 23:06:38.040856 1 manager.go:148] objectbucket.io/provisioner-manager "msg"="stopping provisioner" "name"="rook-ceph.ceph.rook.io/bucket" "reason"={} 2022-03-08T23:06:38.058 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:38.040904 I | op-osd: stopping monitoring of OSDs in namespace "rook-ceph" 2022-03-08T23:06:38.058 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:38.040918 I | op-mon: stopping monitoring of mons in namespace "rook-ceph" 2022-03-08T23:06:38.059 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:38.040929 I | ceph-cluster-controller: stopping monitoring of ceph status 2022-03-08T23:06:38.059 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:38.041038 I | operator: successfully started the controller-runtime manager 2022-03-08T23:06:38.060 INFO:tasks.rook:Waiting for 12 OSDs 2022-03-08T23:06:38.061 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph osd stat -f json 2022-03-08T23:06:38.070 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:38.075511 I | op-k8sutil: ROOK_CURRENT_NAMESPACE_ONLY="false" (env var) 2022-03-08T23:06:38.070 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:38.075531 I | operator: watching all namespaces for Ceph CRs 2022-03-08T23:06:38.070 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:38.075556 I | operator: setting up schemes 2022-03-08T23:06:38.073 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:38.077983 I | operator: setting up the controller-runtime manager 2022-03-08T23:06:38.277 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:38.282504 I | operator: looking for admission webhook secret "rook-ceph-admission-controller" 2022-03-08T23:06:38.282 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:38.285437 I | operator: admission webhook secret "rook-ceph-admission-controller" not found. proceeding without the admission controller 2022-03-08T23:06:38.282 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:38.285472 I | ceph-cluster-controller: successfully started 2022-03-08T23:06:38.282 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:38.285537 I | ceph-cluster-controller: enabling hotplug orchestration 2022-03-08T23:06:38.283 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:38.285566 I | ceph-crashcollector-controller: successfully started 2022-03-08T23:06:38.283 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:38.285601 I | ceph-block-pool-controller: successfully started 2022-03-08T23:06:38.283 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:38.285632 I | ceph-object-store-user-controller: successfully started 2022-03-08T23:06:38.284 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:38.285657 I | ceph-object-realm-controller: successfully started 2022-03-08T23:06:38.284 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:38.285679 I | ceph-object-zonegroup-controller: successfully started 2022-03-08T23:06:38.284 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:38.285704 I | ceph-object-zone-controller: successfully started 2022-03-08T23:06:38.284 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:38.285890 I | ceph-object-controller: successfully started 2022-03-08T23:06:38.284 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:38.285929 I | ceph-file-controller: successfully started 2022-03-08T23:06:38.285 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:38.285973 I | ceph-nfs-controller: successfully started 2022-03-08T23:06:38.285 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:38.286023 I | ceph-rbd-mirror-controller: successfully started 2022-03-08T23:06:38.286 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:38.286057 I | ceph-client-controller: successfully started 2022-03-08T23:06:38.286 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:38.286080 I | ceph-filesystem-mirror-controller: successfully started 2022-03-08T23:06:38.286 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:38.286116 I | operator: rook-ceph-operator-config-controller successfully started 2022-03-08T23:06:38.286 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:38.286141 I | ceph-csi: rook-ceph-operator-csi-controller successfully started 2022-03-08T23:06:38.287 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:38.286179 I | op-bucket-prov: rook-ceph-operator-bucket-controller successfully started 2022-03-08T23:06:38.287 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:38.286212 I | ceph-bucket-topic: successfully started 2022-03-08T23:06:38.287 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:38.286235 I | ceph-bucket-notification: successfully started 2022-03-08T23:06:38.287 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:38.286263 I | ceph-bucket-notification: successfully started 2022-03-08T23:06:38.287 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:38.286284 I | ceph-fs-subvolumegroup-controller: successfully started 2022-03-08T23:06:38.288 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:38.287358 I | operator: starting the controller-runtime manager 2022-03-08T23:06:38.387 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:38.392697 I | op-k8sutil: ROOK_CEPH_COMMANDS_TIMEOUT_SECONDS="15" (configmap) 2022-03-08T23:06:38.387 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:38.392728 I | op-k8sutil: ROOK_LOG_LEVEL="INFO" (configmap) 2022-03-08T23:06:38.387 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:38.392738 I | op-k8sutil: ROOK_ENABLE_DISCOVERY_DAEMON="false" (configmap) 2022-03-08T23:06:38.390 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:38.392932 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-03-08T23:06:38.390 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:38.396134 I | op-mon: parsing mon endpoints: a=10.110.141.123:6789,b=10.98.41.230:6789,c=10.109.47.142:6789 2022-03-08T23:06:38.391 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:38.396192 I | op-k8sutil: ROOK_OBC_WATCH_OPERATOR_NAMESPACE="true" (configmap) 2022-03-08T23:06:38.391 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:38.396200 I | op-bucket-prov: ceph bucket provisioner launched watching for provisioner "rook-ceph.ceph.rook.io/bucket" 2022-03-08T23:06:38.393 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:38.396427 I | operator: rook-ceph-operator-config-controller done reconciling 2022-03-08T23:06:38.393 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:38.397178 I | op-bucket-prov: successfully reconciled bucket provisioner 2022-03-08T23:06:38.394 INFO:tasks.rook.operator.smithi077.stdout:I0308 23:06:38.397222 1 manager.go:135] objectbucket.io/provisioner-manager "msg"="starting provisioner" "name"="rook-ceph.ceph.rook.io/bucket" 2022-03-08T23:06:38.394 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:38.397778 I | op-mon: parsing mon endpoints: a=10.110.141.123:6789,b=10.98.41.230:6789,c=10.109.47.142:6789 2022-03-08T23:06:38.427 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:38.432842 I | ceph-spec: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3... 2022-03-08T23:06:38.457 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:38.463016 I | ceph-csi: successfully created csi config map "rook-ceph-csi-config" 2022-03-08T23:06:38.463 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:38.468891 I | op-k8sutil: ROOK_CSI_ENABLE_RBD="true" (configmap) 2022-03-08T23:06:38.463 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:38.468918 I | op-k8sutil: ROOK_CSI_ENABLE_CEPHFS="true" (configmap) 2022-03-08T23:06:38.463 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:38.468926 I | op-k8sutil: ROOK_CSI_ALLOW_UNSUPPORTED_VERSION="false" (configmap) 2022-03-08T23:06:38.464 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:38.468932 I | op-k8sutil: ROOK_CSI_ENABLE_GRPC_METRICS="false" (configmap) 2022-03-08T23:06:38.466 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:38.468939 I | op-k8sutil: CSI_ENABLE_HOST_NETWORK="true" (default) 2022-03-08T23:06:38.467 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:38.468948 I | op-k8sutil: ROOK_CSI_CEPH_IMAGE="quay.io/cephcsi/cephcsi:v3.5.1" (default) 2022-03-08T23:06:38.467 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:38.468962 I | op-k8sutil: ROOK_CSI_REGISTRAR_IMAGE="k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.5.0" (default) 2022-03-08T23:06:38.467 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:38.468976 I | op-k8sutil: ROOK_CSI_PROVISIONER_IMAGE="k8s.gcr.io/sig-storage/csi-provisioner:v3.1.0" (default) 2022-03-08T23:06:38.467 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:38.468988 I | op-k8sutil: ROOK_CSI_ATTACHER_IMAGE="k8s.gcr.io/sig-storage/csi-attacher:v3.4.0" (default) 2022-03-08T23:06:38.468 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:38.469000 I | op-k8sutil: ROOK_CSI_SNAPSHOTTER_IMAGE="k8s.gcr.io/sig-storage/csi-snapshotter:v5.0.1" (default) 2022-03-08T23:06:38.468 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:38.469011 I | op-k8sutil: ROOK_CSI_KUBELET_DIR_PATH="/var/lib/kubelet" (default) 2022-03-08T23:06:38.468 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:38.469023 I | op-k8sutil: CSI_VOLUME_REPLICATION_IMAGE="quay.io/csiaddons/volumereplication-operator:v0.3.0" (default) 2022-03-08T23:06:38.469 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:38.469035 I | op-k8sutil: ROOK_CSIADDONS_IMAGE="quay.io/csiaddons/k8s-sidecar:v0.2.1" (default) 2022-03-08T23:06:38.469 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:38.469045 I | op-k8sutil: ROOK_CSI_CEPHFS_POD_LABELS="" (default) 2022-03-08T23:06:38.469 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:38.469055 I | op-k8sutil: ROOK_CSI_RBD_POD_LABELS="" (default) 2022-03-08T23:06:38.469 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:38.469068 I | ceph-csi: detecting the ceph csi image version for image "quay.io/cephcsi/cephcsi:v3.5.1" 2022-03-08T23:06:38.469 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:38.469156 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-03-08T23:06:38.470 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:38.469176 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-08T23:06:38.624 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-03-08T23:06:38.645 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":2,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2022-03-08T23:06:38.645 INFO:tasks.rook: have 0/12 OSDs 2022-03-08T23:06:43.767 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:43.771266 I | ceph-csi: Detected ceph CSI image version: "v3.5.1" 2022-03-08T23:06:43.767 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:43.771294 I | op-k8sutil: CSI_FORCE_CEPHFS_KERNEL_CLIENT="true" (configmap) 2022-03-08T23:06:43.767 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:43.771302 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default) 2022-03-08T23:06:43.768 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:43.771307 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default) 2022-03-08T23:06:43.768 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:43.771365 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default) 2022-03-08T23:06:43.768 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:43.771374 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default) 2022-03-08T23:06:43.768 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:43.771382 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default) 2022-03-08T23:06:43.769 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:43.771389 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default) 2022-03-08T23:06:43.769 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:43.771394 I | op-k8sutil: CSIADDONS_PORT="9070" (default) 2022-03-08T23:06:43.769 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:43.771399 I | op-k8sutil: CSIADDONS_PORT="9070" (default) 2022-03-08T23:06:43.769 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:43.771404 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default) 2022-03-08T23:06:43.770 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:43.771409 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default) 2022-03-08T23:06:43.770 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:43.771414 I | op-k8sutil: CSI_PLUGIN_PRIORITY_CLASSNAME="" (default) 2022-03-08T23:06:43.770 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:43.771420 I | op-k8sutil: CSI_PROVISIONER_PRIORITY_CLASSNAME="" (default) 2022-03-08T23:06:43.770 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:43.771424 I | op-k8sutil: CSI_ENABLE_OMAP_GENERATOR="false" (default) 2022-03-08T23:06:43.770 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:43.771430 I | op-k8sutil: CSI_ENABLE_RBD_SNAPSHOTTER="true" (configmap) 2022-03-08T23:06:43.771 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:43.771439 I | op-k8sutil: CSI_ENABLE_CEPHFS_SNAPSHOTTER="true" (configmap) 2022-03-08T23:06:43.771 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:43.771446 I | op-k8sutil: CSI_ENABLE_VOLUME_REPLICATION="false" (configmap) 2022-03-08T23:06:43.771 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:43.771454 I | op-k8sutil: CSI_ENABLE_CSIADDONS="false" (configmap) 2022-03-08T23:06:43.771 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:43.771463 I | op-k8sutil: CSI_CEPHFS_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-03-08T23:06:43.772 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:43.771471 I | op-k8sutil: CSI_RBD_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-03-08T23:06:43.772 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:43.771478 I | op-k8sutil: CSI_PLUGIN_ENABLE_SELINUX_HOST_MOUNT="false" (configmap) 2022-03-08T23:06:43.772 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:43.771487 I | ceph-csi: Kubernetes version is 1.23 2022-03-08T23:06:43.772 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:43.771496 I | op-k8sutil: ROOK_CSI_RESIZER_IMAGE="k8s.gcr.io/sig-storage/csi-resizer:v1.4.0" (default) 2022-03-08T23:06:43.772 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:43.771504 I | op-k8sutil: CSI_LOG_LEVEL="" (default) 2022-03-08T23:06:43.795 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:43.798833 I | op-k8sutil: CSI_PROVISIONER_REPLICAS="2" (configmap) 2022-03-08T23:06:43.800 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:43.806696 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-03-08T23:06:43.801 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:43.806715 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-08T23:06:43.801 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:43.806720 I | op-k8sutil: CSI_PLUGIN_TOLERATIONS="" (default) 2022-03-08T23:06:43.801 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:43.806723 I | op-k8sutil: CSI_PLUGIN_NODE_AFFINITY="" (default) 2022-03-08T23:06:43.801 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:43.806726 I | op-k8sutil: CSI_RBD_PLUGIN_TOLERATIONS="" (default) 2022-03-08T23:06:43.802 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:43.806729 I | op-k8sutil: CSI_RBD_PLUGIN_NODE_AFFINITY="" (default) 2022-03-08T23:06:43.802 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:43.806733 I | op-k8sutil: CSI_RBD_PLUGIN_RESOURCE="" (default) 2022-03-08T23:06:43.947 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:43.953277 I | op-k8sutil: CSI_RBD_PROVISIONER_TOLERATIONS="" (default) 2022-03-08T23:06:43.948 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:43.953306 I | op-k8sutil: CSI_RBD_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-08T23:06:43.948 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:43.953314 I | op-k8sutil: CSI_RBD_PROVISIONER_RESOURCE="" (default) 2022-03-08T23:06:44.400 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:44.406010 I | ceph-spec: detected ceph image version: "17.0.0-11025 quincy" 2022-03-08T23:06:44.401 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:44.406041 I | ceph-cluster-controller: validating ceph version from provided image 2022-03-08T23:06:44.401 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:44.406054 W | ceph-cluster-controller: unsupported ceph version detected: "17.0.0-11025 quincy", pursuing 2022-03-08T23:06:44.610 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:44.615440 I | ceph-csi: successfully started CSI Ceph RBD driver 2022-03-08T23:06:44.700 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:44.706340 I | op-mon: parsing mon endpoints: a=10.110.141.123:6789,b=10.98.41.230:6789,c=10.109.47.142:6789 2022-03-08T23:06:44.776 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:44.782291 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-08T23:06:44.781 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:44.782609 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-08T23:06:44.854 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:44.860098 I | op-k8sutil: CSI_CEPHFS_PLUGIN_TOLERATIONS="" (default) 2022-03-08T23:06:44.855 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:44.860129 I | op-k8sutil: CSI_CEPHFS_PLUGIN_NODE_AFFINITY="" (default) 2022-03-08T23:06:44.855 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:44.860139 I | op-k8sutil: CSI_CEPHFS_PLUGIN_RESOURCE="" (default) 2022-03-08T23:06:44.949 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:44.955408 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_TOLERATIONS="" (default) 2022-03-08T23:06:44.949 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:44.955435 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-08T23:06:44.950 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:44.955443 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_RESOURCE="" (default) 2022-03-08T23:06:45.030 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:45.036438 I | ceph-csi: successfully started CSI CephFS driver 2022-03-08T23:06:45.221 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:45.226596 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-11025 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3" 2022-03-08T23:06:45.296 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:45.302404 I | op-k8sutil: CSI_RBD_FSGROUPPOLICY="ReadWriteOnceWithFSType" (configmap) 2022-03-08T23:06:45.301 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:45.307251 I | ceph-csi: CSIDriver object updated for driver "rook-ceph.rbd.csi.ceph.com" 2022-03-08T23:06:45.301 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:45.307281 I | op-k8sutil: CSI_CEPHFS_FSGROUPPOLICY="ReadWriteOnceWithFSType" (configmap) 2022-03-08T23:06:45.308 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:45.311585 I | ceph-csi: CSIDriver object updated for driver "rook-ceph.cephfs.csi.ceph.com" 2022-03-08T23:06:45.367 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:45.374032 I | op-mon: start running mons 2022-03-08T23:06:45.754 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:45.760131 I | op-mon: parsing mon endpoints: a=10.110.141.123:6789,b=10.98.41.230:6789,c=10.109.47.142:6789 2022-03-08T23:06:46.154 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:46.160293 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.110.141.123:6789","10.98.41.230:6789","10.109.47.142:6789"]}] data:a=10.110.141.123:6789,b=10.98.41.230:6789,c=10.109.47.142:6789 mapping:{"node":{"a":{"Name":"smithi135","Hostname":"smithi135","Address":"172.21.15.135"},"b":{"Name":"smithi106","Hostname":"smithi106","Address":"172.21.15.106"},"c":{"Name":"smithi077","Hostname":"smithi077","Address":"172.21.15.77"}}} maxMonId:2] 2022-03-08T23:06:46.753 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:46.758805 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-08T23:06:46.753 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:46.759060 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-08T23:06:48.356 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:48.361733 I | op-mon: targeting the mon count 3 2022-03-08T23:06:48.361 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:48.367298 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-08T23:06:48.647 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph osd stat -f json 2022-03-08T23:06:48.706 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:48.712930 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-08T23:06:48.707 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:48.712961 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-03-08T23:06:49.075 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:49.080765 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-03-08T23:06:49.075 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:49.080802 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-08T23:06:49.142 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-03-08T23:06:49.165 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":2,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2022-03-08T23:06:49.166 INFO:tasks.rook: have 0/12 OSDs 2022-03-08T23:06:49.411 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:49.417288 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-08T23:06:49.412 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:49.417339 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-03-08T23:06:49.749 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:49.755239 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-03-08T23:06:49.750 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:49.755269 I | op-config: deleting "log file" option from the mon configuration database 2022-03-08T23:06:50.073 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:50.078711 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-03-08T23:06:50.073 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:50.078735 I | op-mon: checking for basic quorum with existing mons 2022-03-08T23:06:50.205 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:50.211057 I | op-mon: mon "a" endpoint is [v2:10.110.141.123:3300,v1:10.110.141.123:6789] 2022-03-08T23:06:50.348 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:50.354088 I | op-mon: mon "b" endpoint is [v2:10.98.41.230:3300,v1:10.98.41.230:6789] 2022-03-08T23:06:50.768 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:50.774138 I | op-mon: mon "c" endpoint is [v2:10.109.47.142:3300,v1:10.109.47.142:6789] 2022-03-08T23:06:51.354 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:51.360209 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.110.141.123:6789","10.98.41.230:6789","10.109.47.142:6789"]}] data:a=10.110.141.123:6789,b=10.98.41.230:6789,c=10.109.47.142:6789 mapping:{"node":{"a":{"Name":"smithi135","Hostname":"smithi135","Address":"172.21.15.135"},"b":{"Name":"smithi106","Hostname":"smithi106","Address":"172.21.15.106"},"c":{"Name":"smithi077","Hostname":"smithi077","Address":"172.21.15.77"}}} maxMonId:2] 2022-03-08T23:06:51.953 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:51.958963 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-08T23:06:51.953 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:51.959226 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-08T23:06:52.374 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:52.380874 I | op-mon: deployment for mon rook-ceph-mon-a already exists. updating if needed 2022-03-08T23:06:52.386 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:52.391939 I | op-k8sutil: deployment "rook-ceph-mon-a" did not change, nothing to update 2022-03-08T23:06:52.386 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:52.391968 I | op-mon: waiting for mon quorum with [a b c] 2022-03-08T23:06:52.957 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:52.963697 I | op-mon: mons running: [a b c] 2022-03-08T23:06:53.366 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:53.371955 I | op-mon: Monitors in quorum: [a b c] 2022-03-08T23:06:53.374 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:53.380974 I | op-mon: deployment for mon rook-ceph-mon-b already exists. updating if needed 2022-03-08T23:06:53.385 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:53.391517 I | op-k8sutil: deployment "rook-ceph-mon-b" did not change, nothing to update 2022-03-08T23:06:53.385 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:53.391543 I | op-mon: waiting for mon quorum with [a b c] 2022-03-08T23:06:53.558 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:53.564278 I | op-mon: mons running: [a b c] 2022-03-08T23:06:53.979 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:53.985116 I | op-mon: Monitors in quorum: [a b c] 2022-03-08T23:06:53.986 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:53.993228 I | op-mon: deployment for mon rook-ceph-mon-c already exists. updating if needed 2022-03-08T23:06:53.993 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:53.999598 I | op-k8sutil: deployment "rook-ceph-mon-c" did not change, nothing to update 2022-03-08T23:06:53.993 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:53.999613 I | op-mon: waiting for mon quorum with [a b c] 2022-03-08T23:06:54.157 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:54.163500 I | op-mon: mons running: [a b c] 2022-03-08T23:06:54.581 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:54.587883 I | op-mon: Monitors in quorum: [a b c] 2022-03-08T23:06:54.582 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:54.587906 I | op-mon: mons created: 3 2022-03-08T23:06:54.948 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:54.954339 I | op-mon: waiting for mon quorum with [a b c] 2022-03-08T23:06:55.039 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:55.045189 I | op-mon: mons running: [a b c] 2022-03-08T23:06:55.437 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:55.443168 I | op-mon: Monitors in quorum: [a b c] 2022-03-08T23:06:55.437 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:55.443211 I | cephclient: getting or creating ceph auth key "client.csi-rbd-provisioner" 2022-03-08T23:06:55.851 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:55.856881 I | cephclient: getting or creating ceph auth key "client.csi-rbd-node" 2022-03-08T23:06:56.267 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:56.273107 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-provisioner" 2022-03-08T23:06:56.681 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:56.687829 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-node" 2022-03-08T23:06:57.091 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:57.097085 I | ceph-csi: created kubernetes csi secrets for cluster "rook-ceph" 2022-03-08T23:06:57.091 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:57.097115 I | cephclient: getting or creating ceph auth key "client.crash" 2022-03-08T23:06:57.488 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:57.494276 I | ceph-crashcollector-controller: created kubernetes crash collector secret for cluster "rook-ceph" 2022-03-08T23:06:57.862 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:57.868274 I | cephclient: successfully enabled msgr2 protocol 2022-03-08T23:06:57.862 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:57.868309 I | cephclient: create rbd-mirror bootstrap peer token "client.rbd-mirror-peer" 2022-03-08T23:06:57.863 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:57.868318 I | cephclient: getting or creating ceph auth key "client.rbd-mirror-peer" 2022-03-08T23:06:58.313 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:58.319516 I | cephclient: successfully created rbd-mirror bootstrap peer token for cluster "rook-ceph" 2022-03-08T23:06:58.380 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:58.386902 I | op-mgr: start running mgr 2022-03-08T23:06:58.381 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:58.386933 I | cephclient: getting or creating ceph auth key "mgr.a" 2022-03-08T23:06:58.870 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:58.876454 I | op-mgr: deployment for mgr rook-ceph-mgr-a already exists. updating if needed 2022-03-08T23:06:58.881 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:58.888045 I | op-k8sutil: deployment "rook-ceph-mgr-a" did not change, nothing to update 2022-03-08T23:06:58.885 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:58.891746 I | op-mgr: setting services to point to mgr "a" 2022-03-08T23:06:59.079 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:59.085480 I | op-mgr: no need to update service "rook-ceph-mgr" 2022-03-08T23:06:59.079 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:59.085559 I | op-mgr: successful modules: balancer 2022-03-08T23:06:59.090 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:59.097148 I | op-osd: start running osds in namespace "rook-ceph" 2022-03-08T23:06:59.091 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:59.097183 I | op-osd: wait timeout for healthy OSDs during upgrade or restart is "10m0s" 2022-03-08T23:06:59.113 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:59.119816 I | op-osd: start provisioning the OSDs on PVCs, if needed 2022-03-08T23:06:59.167 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph osd stat -f json 2022-03-08T23:06:59.271 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:59.276779 I | op-osd: creating 1 new PVCs for device set "smithi077-lv-1" 2022-03-08T23:06:59.475 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:59.481147 I | op-osd: successfully provisioned PVC "smithi077-lv-1-data-0hb6kc" 2022-03-08T23:06:59.475 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:59.481182 I | op-osd: creating 1 new PVCs for device set "smithi077-lv-2" 2022-03-08T23:06:59.673 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:59.679163 I | op-osd: successfully provisioned PVC "smithi077-lv-2-data-0hqxxt" 2022-03-08T23:06:59.766 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:59.679199 I | op-osd: creating 1 new PVCs for device set "smithi077-lv-3" 2022-03-08T23:06:59.767 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-03-08T23:06:59.778 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":2,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2022-03-08T23:06:59.779 INFO:tasks.rook: have 0/12 OSDs 2022-03-08T23:06:59.872 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:59.878721 I | op-osd: successfully provisioned PVC "smithi077-lv-3-data-0zvk2m" 2022-03-08T23:06:59.873 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:06:59.878760 I | op-osd: creating 1 new PVCs for device set "smithi077-lv-4" 2022-03-08T23:07:00.071 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:07:00.077692 I | op-osd: successfully provisioned PVC "smithi077-lv-4-data-0lbfgs" 2022-03-08T23:07:00.072 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:07:00.077734 I | op-osd: creating 1 new PVCs for device set "smithi106-lv-1" 2022-03-08T23:07:00.272 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:07:00.278988 I | op-osd: successfully provisioned PVC "smithi106-lv-1-data-0j5llj" 2022-03-08T23:07:00.273 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:07:00.279029 I | op-osd: creating 1 new PVCs for device set "smithi106-lv-2" 2022-03-08T23:07:00.354 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:07:00.360572 I | op-mgr: successful modules: dashboard 2022-03-08T23:07:00.355 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:07:00.360789 I | op-mgr: successful modules: mgr module(s) from the spec 2022-03-08T23:07:00.357 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:07:00.363659 I | op-mgr: successful modules: prometheus 2022-03-08T23:07:00.492 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:07:00.498375 I | op-osd: successfully provisioned PVC "smithi106-lv-2-data-0r5dlw" 2022-03-08T23:07:00.493 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:07:00.498418 I | op-osd: creating 1 new PVCs for device set "smithi106-lv-3" 2022-03-08T23:07:00.672 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:07:00.678148 I | op-osd: successfully provisioned PVC "smithi106-lv-3-data-07lcs6" 2022-03-08T23:07:00.672 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:07:00.678184 I | op-osd: creating 1 new PVCs for device set "smithi106-lv-4" 2022-03-08T23:07:00.892 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:07:00.898557 I | op-osd: successfully provisioned PVC "smithi106-lv-4-data-04hgzm" 2022-03-08T23:07:00.892 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:07:00.898591 I | op-osd: creating 1 new PVCs for device set "smithi135-lv-1" 2022-03-08T23:07:01.072 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:07:01.078560 I | op-osd: successfully provisioned PVC "smithi135-lv-1-data-0cq5dd" 2022-03-08T23:07:01.072 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:07:01.078601 I | op-osd: creating 1 new PVCs for device set "smithi135-lv-2" 2022-03-08T23:07:01.273 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:07:01.278975 I | op-osd: successfully provisioned PVC "smithi135-lv-2-data-0jcrvg" 2022-03-08T23:07:01.273 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:07:01.279013 I | op-osd: creating 1 new PVCs for device set "smithi135-lv-3" 2022-03-08T23:07:01.471 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:07:01.477122 I | op-osd: successfully provisioned PVC "smithi135-lv-3-data-0d9nsm" 2022-03-08T23:07:01.471 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:07:01.477159 I | op-osd: creating 1 new PVCs for device set "smithi135-lv-4" 2022-03-08T23:07:01.683 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:07:01.690101 I | op-osd: successfully provisioned PVC "smithi135-lv-4-data-07k7pp" 2022-03-08T23:07:01.714 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:07:01.721313 I | op-osd: OSD will have its main bluestore block on "smithi077-lv-1-data-0hb6kc" 2022-03-08T23:07:01.860 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:07:01.867024 I | op-mgr: successful modules: orchestrator modules 2022-03-08T23:07:02.095 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:07:02.101994 I | op-osd: started OSD provisioning job for PVC "smithi077-lv-1-data-0hb6kc" 2022-03-08T23:07:02.096 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:07:02.102020 I | op-osd: OSD will have its main bluestore block on "smithi077-lv-2-data-0hqxxt" 2022-03-08T23:07:02.498 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:07:02.504102 I | op-osd: started OSD provisioning job for PVC "smithi077-lv-2-data-0hqxxt" 2022-03-08T23:07:02.498 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:07:02.504139 I | op-osd: OSD will have its main bluestore block on "smithi077-lv-3-data-0zvk2m" 2022-03-08T23:07:02.887 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:07:02.893213 I | op-osd: started OSD provisioning job for PVC "smithi077-lv-3-data-0zvk2m" 2022-03-08T23:07:02.887 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:07:02.893249 I | op-osd: OSD will have its main bluestore block on "smithi077-lv-4-data-0lbfgs" 2022-03-08T23:07:03.347 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:07:03.353292 I | op-osd: started OSD provisioning job for PVC "smithi077-lv-4-data-0lbfgs" 2022-03-08T23:07:03.347 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:07:03.353328 I | op-osd: OSD will have its main bluestore block on "smithi106-lv-1-data-0j5llj" 2022-03-08T23:07:03.926 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:07:03.932663 I | op-osd: started OSD provisioning job for PVC "smithi106-lv-1-data-0j5llj" 2022-03-08T23:07:03.927 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:07:03.932697 I | op-osd: OSD will have its main bluestore block on "smithi106-lv-2-data-0r5dlw" 2022-03-08T23:07:04.175 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:07:04.181375 I | op-osd: started OSD provisioning job for PVC "smithi106-lv-2-data-0r5dlw" 2022-03-08T23:07:04.175 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:07:04.181409 I | op-osd: OSD will have its main bluestore block on "smithi106-lv-3-data-07lcs6" 2022-03-08T23:07:04.632 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:07:04.638452 I | op-osd: started OSD provisioning job for PVC "smithi106-lv-3-data-07lcs6" 2022-03-08T23:07:04.632 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:07:04.638508 I | op-osd: OSD will have its main bluestore block on "smithi106-lv-4-data-04hgzm" 2022-03-08T23:07:04.940 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:07:04.946427 I | op-osd: started OSD provisioning job for PVC "smithi106-lv-4-data-04hgzm" 2022-03-08T23:07:04.940 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:07:04.946466 I | op-osd: OSD will have its main bluestore block on "smithi135-lv-1-data-0cq5dd" 2022-03-08T23:07:05.286 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:07:05.292681 I | op-osd: started OSD provisioning job for PVC "smithi135-lv-1-data-0cq5dd" 2022-03-08T23:07:05.286 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:07:05.292718 I | op-osd: OSD will have its main bluestore block on "smithi135-lv-2-data-0jcrvg" 2022-03-08T23:07:05.737 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:07:05.743888 I | op-osd: started OSD provisioning job for PVC "smithi135-lv-2-data-0jcrvg" 2022-03-08T23:07:05.738 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:07:05.743926 I | op-osd: OSD will have its main bluestore block on "smithi135-lv-3-data-0d9nsm" 2022-03-08T23:07:06.424 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:07:06.431008 I | op-osd: started OSD provisioning job for PVC "smithi135-lv-3-data-0d9nsm" 2022-03-08T23:07:06.425 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:07:06.431040 I | op-osd: OSD will have its main bluestore block on "smithi135-lv-4-data-07k7pp" 2022-03-08T23:07:06.739 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:07:06.745158 I | op-osd: started OSD provisioning job for PVC "smithi135-lv-4-data-07k7pp" 2022-03-08T23:07:06.739 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:07:06.745199 I | op-osd: start provisioning the OSDs on nodes, if needed 2022-03-08T23:07:06.739 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:07:06.745207 I | op-osd: no nodes are defined for configuring OSDs on raw devices 2022-03-08T23:07:06.973 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:07:06.979704 I | op-osd: OSD orchestration status for PVC smithi077-lv-1-data-0hb6kc is "starting" 2022-03-08T23:07:06.974 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:07:06.979729 I | op-osd: OSD orchestration status for PVC smithi077-lv-2-data-0hqxxt is "starting" 2022-03-08T23:07:06.974 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:07:06.979735 I | op-osd: OSD orchestration status for PVC smithi077-lv-3-data-0zvk2m is "starting" 2022-03-08T23:07:06.974 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:07:06.979740 I | op-osd: OSD orchestration status for PVC smithi077-lv-4-data-0lbfgs is "starting" 2022-03-08T23:07:06.974 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:07:06.979745 I | op-osd: OSD orchestration status for PVC smithi106-lv-1-data-0j5llj is "starting" 2022-03-08T23:07:06.975 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:07:06.979750 I | op-osd: OSD orchestration status for PVC smithi106-lv-2-data-0r5dlw is "starting" 2022-03-08T23:07:06.975 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:07:06.979758 I | op-osd: OSD orchestration status for PVC smithi106-lv-3-data-07lcs6 is "starting" 2022-03-08T23:07:06.975 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:07:06.979767 I | op-osd: OSD orchestration status for PVC smithi106-lv-4-data-04hgzm is "starting" 2022-03-08T23:07:06.975 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:07:06.979776 I | op-osd: OSD orchestration status for PVC smithi135-lv-1-data-0cq5dd is "starting" 2022-03-08T23:07:06.975 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:07:06.979784 I | op-osd: OSD orchestration status for PVC smithi135-lv-2-data-0jcrvg is "starting" 2022-03-08T23:07:06.976 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:07:06.979789 I | op-osd: OSD orchestration status for PVC smithi135-lv-3-data-0d9nsm is "starting" 2022-03-08T23:07:06.976 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:07:06.979797 I | op-osd: OSD orchestration status for PVC smithi135-lv-4-data-07k7pp is "starting" 2022-03-08T23:07:09.780 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph osd stat -f json 2022-03-08T23:07:10.920 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-03-08T23:07:10.938 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":2,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2022-03-08T23:07:10.938 INFO:tasks.rook: have 0/12 OSDs 2022-03-08T23:07:16.011 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:07:16.017799 I | op-osd: OSD orchestration status for node smithi077-lv-3-data-0zvk2m is "orchestrating" 2022-03-08T23:07:16.278 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:07:16.285477 I | op-osd: OSD orchestration status for node smithi106-lv-2-data-0r5dlw is "orchestrating" 2022-03-08T23:07:16.473 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:07:16.480341 I | op-osd: OSD orchestration status for PVC smithi077-lv-3-data-0zvk2m is "orchestrating" 2022-03-08T23:07:16.685 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:07:16.692260 I | op-osd: OSD orchestration status for PVC smithi106-lv-2-data-0r5dlw is "orchestrating" 2022-03-08T23:07:19.065 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:07:19.071740 I | op-osd: OSD orchestration status for node smithi106-lv-4-data-04hgzm is "orchestrating" 2022-03-08T23:07:19.664 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:07:19.670977 I | op-osd: OSD orchestration status for PVC smithi106-lv-4-data-04hgzm is "orchestrating" 2022-03-08T23:07:20.785 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:07:20.791951 I | op-osd: OSD orchestration status for node smithi135-lv-3-data-0d9nsm is "orchestrating" 2022-03-08T23:07:20.939 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph osd stat -f json 2022-03-08T23:07:21.021 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:07:21.028187 I | op-osd: OSD orchestration status for node smithi135-lv-4-data-07k7pp is "orchestrating" 2022-03-08T23:07:21.407 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:07:21.414549 I | op-osd: OSD orchestration status for PVC smithi135-lv-3-data-0d9nsm is "orchestrating" 2022-03-08T23:07:21.867 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:07:21.873231 I | op-osd: OSD orchestration status for PVC smithi135-lv-4-data-07k7pp is "orchestrating" 2022-03-08T23:07:22.413 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-03-08T23:07:22.435 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":4,"num_osds":2,"num_up_osds":0,"osd_up_since":0,"num_in_osds":2,"osd_in_since":1646780839,"num_remapped_pgs":0} 2022-03-08T23:07:22.435 INFO:tasks.rook: have 0/12 OSDs 2022-03-08T23:07:24.065 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:07:24.071896 I | op-osd: OSD orchestration status for node smithi077-lv-4-data-0lbfgs is "orchestrating" 2022-03-08T23:07:25.006 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:07:25.012779 I | op-osd: OSD orchestration status for PVC smithi077-lv-4-data-0lbfgs is "orchestrating" 2022-03-08T23:07:25.588 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:07:25.595370 I | op-osd: OSD orchestration status for PVC smithi077-lv-3-data-0zvk2m is "completed" 2022-03-08T23:07:25.589 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:07:25.595397 I | op-osd: creating OSD 0 on PVC "smithi077-lv-3-data-0zvk2m" 2022-03-08T23:07:25.589 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:07:25.595406 I | op-osd: OSD will have its main bluestore block on "smithi077-lv-3-data-0zvk2m" 2022-03-08T23:07:26.709 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:07:26.716007 I | op-osd: OSD orchestration status for node smithi106-lv-3-data-07lcs6 is "orchestrating" 2022-03-08T23:07:26.710 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:07:26.716076 I | op-osd: OSD orchestration status for node smithi106-lv-1-data-0j5llj is "orchestrating" 2022-03-08T23:07:26.715 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:07:26.716813 I | op-osd: OSD orchestration status for node smithi135-lv-2-data-0jcrvg is "orchestrating" 2022-03-08T23:07:27.210 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:07:27.216554 I | op-osd: OSD orchestration status for PVC smithi106-lv-2-data-0r5dlw is "completed" 2022-03-08T23:07:27.210 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:07:27.216595 I | op-osd: creating OSD 1 on PVC "smithi106-lv-2-data-0r5dlw" 2022-03-08T23:07:27.211 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:07:27.216612 I | op-osd: OSD will have its main bluestore block on "smithi106-lv-2-data-0r5dlw" 2022-03-08T23:07:28.376 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:07:28.381252 I | op-osd: OSD orchestration status for PVC smithi106-lv-3-data-07lcs6 is "orchestrating" 2022-03-08T23:07:28.377 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:07:28.381459 I | op-osd: OSD orchestration status for PVC smithi106-lv-1-data-0j5llj is "orchestrating" 2022-03-08T23:07:28.377 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:07:28.381590 I | op-osd: OSD orchestration status for node smithi135-lv-1-data-0cq5dd is "orchestrating" 2022-03-08T23:07:28.377 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:07:28.381704 I | op-osd: OSD orchestration status for PVC smithi135-lv-2-data-0jcrvg is "orchestrating" 2022-03-08T23:07:28.450 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:07:28.450127 I | op-osd: OSD orchestration status for PVC smithi135-lv-1-data-0cq5dd is "orchestrating" 2022-03-08T23:07:28.997 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:07:29.002899 I | op-osd: OSD orchestration status for node smithi077-lv-1-data-0hb6kc is "orchestrating" 2022-03-08T23:07:29.029 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:07:29.035291 I | op-osd: OSD orchestration status for node smithi077-lv-2-data-0hqxxt is "orchestrating" 2022-03-08T23:07:29.374 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:07:29.373296 I | op-osd: OSD orchestration status for PVC smithi106-lv-1-data-0j5llj is "failed" 2022-03-08T23:07:29.374 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:07:29.373329 E | op-osd: failed to provision OSD(s) on PVC smithi106-lv-1-data-0j5llj. &{OSDs:[] Status:failed PvcBackedOSD:true Message:failed to configure devices: failed to initialize devices on PVC: failed to run ceph-volume. Running command: /usr/bin/ceph-authtool --gen-print-key 2022-03-08T23:07:29.375 INFO:tasks.rook.operator.smithi077.stdout:Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 674c3b4d-af5e-4d6b-bb8a-fd0006b214b7 2022-03-08T23:07:29.375 INFO:tasks.rook.operator.smithi077.stdout: stderr: unable to get monitor info from DNS SRV with service name: ceph-mon 2022-03-08T23:07:29.375 INFO:tasks.rook.operator.smithi077.stdout: stderr: 2022-03-08T23:07:29.172+0000 7f66983d5700 -1 failed for service _ceph-mon._tcp 2022-03-08T23:07:29.375 INFO:tasks.rook.operator.smithi077.stdout: stderr: 2022-03-08T23:07:29.172+0000 7f66983d5700 -1 monclient: get_monmap_and_config cannot identify monitors to contact 2022-03-08T23:07:29.376 INFO:tasks.rook.operator.smithi077.stdout: stderr: [errno 2] RADOS object not found (error connecting to the cluster) 2022-03-08T23:07:29.376 INFO:tasks.rook.operator.smithi077.stdout:Traceback (most recent call last): 2022-03-08T23:07:29.376 INFO:tasks.rook.operator.smithi077.stdout: File "/usr/sbin/ceph-volume", line 11, in 2022-03-08T23:07:29.376 INFO:tasks.rook.operator.smithi077.stdout: load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2022-03-08T23:07:29.377 INFO:tasks.rook.operator.smithi077.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2022-03-08T23:07:29.377 INFO:tasks.rook.operator.smithi077.stdout: self.main(self.argv) 2022-03-08T23:07:29.377 INFO:tasks.rook.operator.smithi077.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2022-03-08T23:07:29.377 INFO:tasks.rook.operator.smithi077.stdout: return f(*a, **kw) 2022-03-08T23:07:29.377 INFO:tasks.rook.operator.smithi077.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2022-03-08T23:07:29.378 INFO:tasks.rook.operator.smithi077.stdout: terminal.dispatch(self.mapper, subcommand_args) 2022-03-08T23:07:29.378 INFO:tasks.rook.operator.smithi077.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2022-03-08T23:07:29.378 INFO:tasks.rook.operator.smithi077.stdout: instance.main() 2022-03-08T23:07:29.378 INFO:tasks.rook.operator.smithi077.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/main.py", line 32, in main 2022-03-08T23:07:29.379 INFO:tasks.rook.operator.smithi077.stdout: terminal.dispatch(self.mapper, self.argv) 2022-03-08T23:07:29.379 INFO:tasks.rook.operator.smithi077.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2022-03-08T23:07:29.379 INFO:tasks.rook.operator.smithi077.stdout: instance.main() 2022-03-08T23:07:29.379 INFO:tasks.rook.operator.smithi077.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 169, in main 2022-03-08T23:07:29.379 INFO:tasks.rook.operator.smithi077.stdout: self.safe_prepare(self.args) 2022-03-08T23:07:29.380 INFO:tasks.rook.operator.smithi077.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 91, in safe_prepare 2022-03-08T23:07:29.380 INFO:tasks.rook.operator.smithi077.stdout: self.prepare() 2022-03-08T23:07:29.380 INFO:tasks.rook.operator.smithi077.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2022-03-08T23:07:29.380 INFO:tasks.rook.operator.smithi077.stdout: return func(*a, **kw) 2022-03-08T23:07:29.381 INFO:tasks.rook.operator.smithi077.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 125, in prepare 2022-03-08T23:07:29.381 INFO:tasks.rook.operator.smithi077.stdout: osd_fsid, json.dumps(secrets)) 2022-03-08T23:07:29.381 INFO:tasks.rook.operator.smithi077.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/util/prepare.py", line 173, in create_id 2022-03-08T23:07:29.381 INFO:tasks.rook.operator.smithi077.stdout: raise RuntimeError('Unable to create a new OSD id') 2022-03-08T23:07:29.381 INFO:tasks.rook.operator.smithi077.stdout:RuntimeError: Unable to create a new OSD id. debug logs below: 2022-03-08T23:07:29.382 INFO:tasks.rook.operator.smithi077.stdout:[2022-03-08 23:07:28,584][ceph_volume.main][INFO ] Running command: ceph-volume --log-path /var/log/ceph/smithi106-lv-1-data-0j5llj raw prepare --bluestore --data /dev/mapper/vg_nvme-lv_1 2022-03-08T23:07:29.382 INFO:tasks.rook.operator.smithi077.stdout:[2022-03-08 23:07:28,585][ceph_volume.process][INFO ] Running command: /usr/bin/lsblk -plno KNAME,NAME,TYPE 2022-03-08T23:07:29.382 INFO:tasks.rook.operator.smithi077.stdout:[2022-03-08 23:07:28,592][ceph_volume.process][INFO ] stdout /dev/loop0 /dev/loop0 loop 2022-03-08T23:07:29.382 INFO:tasks.rook.operator.smithi077.stdout:[2022-03-08 23:07:28,592][ceph_volume.process][INFO ] stdout /dev/loop1 /dev/loop1 loop 2022-03-08T23:07:29.383 INFO:tasks.rook.operator.smithi077.stdout:[2022-03-08 23:07:28,592][ceph_volume.process][INFO ] stdout /dev/loop2 /dev/loop2 loop 2022-03-08T23:07:29.383 INFO:tasks.rook.operator.smithi077.stdout:[2022-03-08 23:07:28,592][ceph_volume.process][INFO ] stdout /dev/loop3 /dev/loop3 loop 2022-03-08T23:07:29.383 INFO:tasks.rook.operator.smithi077.stdout:[2022-03-08 23:07:28,593][ceph_volume.process][INFO ] stdout /dev/sda /dev/sda disk 2022-03-08T23:07:29.383 INFO:tasks.rook.operator.smithi077.stdout:[2022-03-08 23:07:28,593][ceph_volume.process][INFO ] stdout /dev/sda1 /dev/sda1 part 2022-03-08T23:07:29.383 INFO:tasks.rook.operator.smithi077.stdout:[2022-03-08 23:07:28,593][ceph_volume.process][INFO ] stdout /dev/dm-0 /dev/mapper/vg_nvme-lv_1 lvm 2022-03-08T23:07:29.384 INFO:tasks.rook.operator.smithi077.stdout:[2022-03-08 23:07:28,593][ceph_volume.process][INFO ] stdout /dev/dm-1 /dev/mapper/vg_nvme-lv_2 lvm 2022-03-08T23:07:29.384 INFO:tasks.rook.operator.smithi077.stdout:[2022-03-08 23:07:28,593][ceph_volume.process][INFO ] stdout /dev/dm-2 /dev/mapper/vg_nvme-lv_3 lvm 2022-03-08T23:07:29.384 INFO:tasks.rook.operator.smithi077.stdout:[2022-03-08 23:07:28,593][ceph_volume.process][INFO ] stdout /dev/dm-3 /dev/mapper/vg_nvme-lv_4 lvm 2022-03-08T23:07:29.384 INFO:tasks.rook.operator.smithi077.stdout:[2022-03-08 23:07:28,593][ceph_volume.process][INFO ] stdout /dev/dm-4 /dev/mapper/vg_nvme-lv_5 lvm 2022-03-08T23:07:29.385 INFO:tasks.rook.operator.smithi077.stdout:[2022-03-08 23:07:28,593][ceph_volume.process][INFO ] stdout /dev/nvme0n1 /dev/nvme0n1 disk 2022-03-08T23:07:29.385 INFO:tasks.rook.operator.smithi077.stdout:[2022-03-08 23:07:28,593][ceph_volume.process][INFO ] stdout /dev/nvme1n1 /dev/nvme1n1 disk 2022-03-08T23:07:29.385 INFO:tasks.rook.operator.smithi077.stdout:[2022-03-08 23:07:28,593][ceph_volume.process][INFO ] stdout /dev/nvme2n1 /dev/nvme2n1 disk 2022-03-08T23:07:29.385 INFO:tasks.rook.operator.smithi077.stdout:[2022-03-08 23:07:28,593][ceph_volume.process][INFO ] stdout /dev/nvme3n1 /dev/nvme3n1 disk 2022-03-08T23:07:29.386 INFO:tasks.rook.operator.smithi077.stdout:[2022-03-08 23:07:28,593][ceph_volume.process][INFO ] stdout /dev/nvme4n1 /dev/nvme4n1 disk 2022-03-08T23:07:29.386 INFO:tasks.rook.operator.smithi077.stdout:[2022-03-08 23:07:28,601][ceph_volume.util.system][WARNING] Executable lvs not found on the host, will return lvs as-is 2022-03-08T23:07:29.386 INFO:tasks.rook.operator.smithi077.stdout:[2022-03-08 23:07:28,601][ceph_volume.process][INFO ] Running command: lvs --noheadings --readonly --separator=";" -a --units=b --nosuffix -S lv_path=/dev/mapper/vg_nvme-lv_1 -o lv_tags,lv_path,lv_name,vg_name,lv_uuid,lv_size 2022-03-08T23:07:29.386 INFO:tasks.rook.operator.smithi077.stdout:[2022-03-08 23:07:29,021][ceph_volume.process][INFO ] Running command: /usr/bin/lsblk --nodeps -P -o NAME,KNAME,MAJ:MIN,FSTYPE,MOUNTPOINT,LABEL,UUID,RO,RM,MODEL,SIZE,STATE,OWNER,GROUP,MODE,ALIGNMENT,PHY-SEC,LOG-SEC,ROTA,SCHED,TYPE,DISC-ALN,DISC-GRAN,DISC-MAX,DISC-ZERO,PKNAME,PARTLABEL /dev/mapper/vg_nvme-lv_1 2022-03-08T23:07:29.386 INFO:tasks.rook.operator.smithi077.stdout:[2022-03-08 23:07:29,029][ceph_volume.process][INFO ] stdout NAME="vg_nvme-lv_1" KNAME="dm-0" MAJ:MIN="253:0" FSTYPE="" MOUNTPOINT="" LABEL="" UUID="" RO="0" RM="0" MODEL="" SIZE="89.4G" STATE="running" OWNER="root" GROUP="disk" MODE="brw-rw----" ALIGNMENT="0" PHY-SEC="512" LOG-SEC="512" ROTA="0" SCHED="" TYPE="lvm" DISC-ALN="0" DISC-GRAN="512B" DISC-MAX="2T" DISC-ZERO="0" PKNAME="" PARTLABEL="" 2022-03-08T23:07:29.387 INFO:tasks.rook.operator.smithi077.stdout:[2022-03-08 23:07:29,029][ceph_volume.process][INFO ] Running command: /usr/sbin/blkid -c /dev/null -p /dev/mapper/vg_nvme-lv_1 2022-03-08T23:07:29.387 INFO:tasks.rook.operator.smithi077.stdout:[2022-03-08 23:07:29,032][ceph_volume.util.disk][INFO ] opening device /dev/mapper/vg_nvme-lv_1 to check for BlueStore label 2022-03-08T23:07:29.387 INFO:tasks.rook.operator.smithi077.stdout:[2022-03-08 23:07:29,032][ceph_volume.util.disk][INFO ] opening device /dev/mapper/vg_nvme-lv_1 to check for BlueStore label 2022-03-08T23:07:29.387 INFO:tasks.rook.operator.smithi077.stdout:[2022-03-08 23:07:29,032][ceph_volume.process][INFO ] Running command: /usr/sbin/udevadm info --query=property /dev/mapper/vg_nvme-lv_1 2022-03-08T23:07:29.388 INFO:tasks.rook.operator.smithi077.stdout:[2022-03-08 23:07:29,045][ceph_volume.process][INFO ] stdout DEVLINKS=/dev/vg_nvme/lv_1 /dev/disk/by-id/dm-uuid-LVM-tnFwFsOQYo0FTayanllK4BxIyJ5MyZU17n1F81A27eTqUDSQa4J8NKnLim07bvel /dev/mapper/vg_nvme-lv_1 /dev/disk/by-id/dm-name-vg_nvme-lv_1 2022-03-08T23:07:29.388 INFO:tasks.rook.operator.smithi077.stdout:[2022-03-08 23:07:29,045][ceph_volume.process][INFO ] stdout DEVNAME=/dev/dm-0 2022-03-08T23:07:29.388 INFO:tasks.rook.operator.smithi077.stdout:[2022-03-08 23:07:29,045][ceph_volume.process][INFO ] stdout DEVPATH=/devices/virtual/block/dm-0 2022-03-08T23:07:29.388 INFO:tasks.rook.operator.smithi077.stdout:[2022-03-08 23:07:29,045][ceph_volume.process][INFO ] stdout DEVTYPE=disk 2022-03-08T23:07:29.388 INFO:tasks.rook.operator.smithi077.stdout:[2022-03-08 23:07:29,046][ceph_volume.process][INFO ] stdout DM_LV_NAME=lv_1 2022-03-08T23:07:29.389 INFO:tasks.rook.operator.smithi077.stdout:[2022-03-08 23:07:29,046][ceph_volume.process][INFO ] stdout DM_NAME=vg_nvme-lv_1 2022-03-08T23:07:29.389 INFO:tasks.rook.operator.smithi077.stdout:[2022-03-08 23:07:29,046][ceph_volume.process][INFO ] stdout DM_SUSPENDED=0 2022-03-08T23:07:29.389 INFO:tasks.rook.operator.smithi077.stdout:[2022-03-08 23:07:29,046][ceph_volume.process][INFO ] stdout DM_UDEV_DISABLE_LIBRARY_FALLBACK_FLAG=1 2022-03-08T23:07:29.389 INFO:tasks.rook.operator.smithi077.stdout:[2022-03-08 23:07:29,046][ceph_volume.process][INFO ] stdout DM_UDEV_PRIMARY_SOURCE_FLAG=1 2022-03-08T23:07:29.389 INFO:tasks.rook.operator.smithi077.stdout:[2022-03-08 23:07:29,046][ceph_volume.process][INFO ] stdout DM_UDEV_RULES=1 2022-03-08T23:07:29.390 INFO:tasks.rook.operator.smithi077.stdout:[2022-03-08 23:07:29,046][ceph_volume.process][INFO ] stdout DM_UDEV_RULES_VSN=2 2022-03-08T23:07:29.390 INFO:tasks.rook.operator.smithi077.stdout:[2022-03-08 23:07:29,046][ceph_volume.process][INFO ] stdout DM_UUID=LVM-tnFwFsOQYo0FTayanllK4BxIyJ5MyZU17n1F81A27eTqUDSQa4J8NKnLim07bvel 2022-03-08T23:07:29.390 INFO:tasks.rook.operator.smithi077.stdout:[2022-03-08 23:07:29,046][ceph_volume.process][INFO ] stdout DM_VG_NAME=vg_nvme 2022-03-08T23:07:29.390 INFO:tasks.rook.operator.smithi077.stdout:[2022-03-08 23:07:29,046][ceph_volume.process][INFO ] stdout MAJOR=253 2022-03-08T23:07:29.391 INFO:tasks.rook.operator.smithi077.stdout:[2022-03-08 23:07:29,046][ceph_volume.process][INFO ] stdout MINOR=0 2022-03-08T23:07:29.391 INFO:tasks.rook.operator.smithi077.stdout:[2022-03-08 23:07:29,046][ceph_volume.process][INFO ] stdout SUBSYSTEM=block 2022-03-08T23:07:29.391 INFO:tasks.rook.operator.smithi077.stdout:[2022-03-08 23:07:29,046][ceph_volume.process][INFO ] stdout TAGS=:systemd: 2022-03-08T23:07:29.391 INFO:tasks.rook.operator.smithi077.stdout:[2022-03-08 23:07:29,046][ceph_volume.process][INFO ] stdout USEC_INITIALIZED=197899962 2022-03-08T23:07:29.391 INFO:tasks.rook.operator.smithi077.stdout:[2022-03-08 23:07:29,046][ceph_volume.process][INFO ] Running command: /usr/bin/ceph-authtool --gen-print-key 2022-03-08T23:07:29.392 INFO:tasks.rook.operator.smithi077.stdout:[2022-03-08 23:07:29,064][ceph_volume.process][INFO ] stdout AQCx4Sdi1/fDAxAAOXt0E8Z5e1UmxUpX//gVvA== 2022-03-08T23:07:29.392 INFO:tasks.rook.operator.smithi077.stdout:[2022-03-08 23:07:29,068][ceph_volume.process][INFO ] Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 674c3b4d-af5e-4d6b-bb8a-fd0006b214b7 2022-03-08T23:07:29.392 INFO:tasks.rook.operator.smithi077.stdout:[2022-03-08 23:07:29,181][ceph_volume.process][INFO ] stderr unable to get monitor info from DNS SRV with service name: ceph-mon 2022-03-08T23:07:29.392 INFO:tasks.rook.operator.smithi077.stdout:[2022-03-08 23:07:29,181][ceph_volume.process][INFO ] stderr 2022-03-08T23:07:29.172+0000 7f66983d5700 -1 failed for service _ceph-mon._tcp 2022-03-08T23:07:29.393 INFO:tasks.rook.operator.smithi077.stdout:[2022-03-08 23:07:29,181][ceph_volume.process][INFO ] stderr 2022-03-08T23:07:29.172+0000 7f66983d5700 -1 monclient: get_monmap_and_config cannot identify monitors to contact 2022-03-08T23:07:29.393 INFO:tasks.rook.operator.smithi077.stdout:[2022-03-08 23:07:29,181][ceph_volume.process][INFO ] stderr [errno 2] RADOS object not found (error connecting to the cluster) 2022-03-08T23:07:29.393 INFO:tasks.rook.operator.smithi077.stdout:[2022-03-08 23:07:29,181][ceph_volume.devices.raw.prepare][ERROR ] raw prepare was unable to complete 2022-03-08T23:07:29.393 INFO:tasks.rook.operator.smithi077.stdout:Traceback (most recent call last): 2022-03-08T23:07:29.393 INFO:tasks.rook.operator.smithi077.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 91, in safe_prepare 2022-03-08T23:07:29.394 INFO:tasks.rook.operator.smithi077.stdout: self.prepare() 2022-03-08T23:07:29.394 INFO:tasks.rook.operator.smithi077.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2022-03-08T23:07:29.394 INFO:tasks.rook.operator.smithi077.stdout: return func(*a, **kw) 2022-03-08T23:07:29.394 INFO:tasks.rook.operator.smithi077.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 125, in prepare 2022-03-08T23:07:29.395 INFO:tasks.rook.operator.smithi077.stdout: osd_fsid, json.dumps(secrets)) 2022-03-08T23:07:29.395 INFO:tasks.rook.operator.smithi077.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/util/prepare.py", line 173, in create_id 2022-03-08T23:07:29.395 INFO:tasks.rook.operator.smithi077.stdout: raise RuntimeError('Unable to create a new OSD id') 2022-03-08T23:07:29.395 INFO:tasks.rook.operator.smithi077.stdout:RuntimeError: Unable to create a new OSD id 2022-03-08T23:07:29.395 INFO:tasks.rook.operator.smithi077.stdout:[2022-03-08 23:07:29,183][ceph_volume.devices.raw.prepare][INFO ] will rollback OSD ID creation 2022-03-08T23:07:29.396 INFO:tasks.rook.operator.smithi077.stdout:[2022-03-08 23:07:29,183][ceph_volume][ERROR ] exception caught by decorator 2022-03-08T23:07:29.396 INFO:tasks.rook.operator.smithi077.stdout:Traceback (most recent call last): 2022-03-08T23:07:29.396 INFO:tasks.rook.operator.smithi077.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2022-03-08T23:07:29.396 INFO:tasks.rook.operator.smithi077.stdout: return f(*a, **kw) 2022-03-08T23:07:29.397 INFO:tasks.rook.operator.smithi077.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2022-03-08T23:07:29.397 INFO:tasks.rook.operator.smithi077.stdout: terminal.dispatch(self.mapper, subcommand_args) 2022-03-08T23:07:29.397 INFO:tasks.rook.operator.smithi077.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2022-03-08T23:07:29.397 INFO:tasks.rook.operator.smithi077.stdout: instance.main() 2022-03-08T23:07:29.397 INFO:tasks.rook.operator.smithi077.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/main.py", line 32, in main 2022-03-08T23:07:29.398 INFO:tasks.rook.operator.smithi077.stdout: terminal.dispatch(self.mapper, self.argv) 2022-03-08T23:07:29.398 INFO:tasks.rook.operator.smithi077.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2022-03-08T23:07:29.398 INFO:tasks.rook.operator.smithi077.stdout: instance.main() 2022-03-08T23:07:29.398 INFO:tasks.rook.operator.smithi077.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 169, in main 2022-03-08T23:07:29.398 INFO:tasks.rook.operator.smithi077.stdout: self.safe_prepare(self.args) 2022-03-08T23:07:29.399 INFO:tasks.rook.operator.smithi077.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 91, in safe_prepare 2022-03-08T23:07:29.399 INFO:tasks.rook.operator.smithi077.stdout: self.prepare() 2022-03-08T23:07:29.399 INFO:tasks.rook.operator.smithi077.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2022-03-08T23:07:29.399 INFO:tasks.rook.operator.smithi077.stdout: return func(*a, **kw) 2022-03-08T23:07:29.400 INFO:tasks.rook.operator.smithi077.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 125, in prepare 2022-03-08T23:07:29.400 INFO:tasks.rook.operator.smithi077.stdout: osd_fsid, json.dumps(secrets)) 2022-03-08T23:07:29.400 INFO:tasks.rook.operator.smithi077.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/util/prepare.py", line 173, in create_id 2022-03-08T23:07:29.400 INFO:tasks.rook.operator.smithi077.stdout: raise RuntimeError('Unable to create a new OSD id') 2022-03-08T23:07:29.400 INFO:tasks.rook.operator.smithi077.stdout:RuntimeError: Unable to create a new OSD id 2022-03-08T23:07:29.401 INFO:tasks.rook.operator.smithi077.stdout:: exit status 1} 2022-03-08T23:07:29.904 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:07:29.910493 I | op-osd: OSD orchestration status for PVC smithi077-lv-1-data-0hb6kc is "orchestrating" 2022-03-08T23:07:29.908 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:07:29.913201 I | op-osd: OSD orchestration status for PVC smithi106-lv-4-data-04hgzm is "completed" 2022-03-08T23:07:29.908 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:07:29.913228 I | op-osd: creating OSD 2 on PVC "smithi106-lv-4-data-04hgzm" 2022-03-08T23:07:29.909 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:07:29.913245 I | op-osd: OSD will have its main bluestore block on "smithi106-lv-4-data-04hgzm" 2022-03-08T23:07:30.333 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:07:30.338960 I | op-osd: OSD orchestration status for PVC smithi077-lv-2-data-0hqxxt is "orchestrating" 2022-03-08T23:07:30.335 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:07:30.339238 I | op-osd: OSD orchestration status for PVC smithi135-lv-3-data-0d9nsm is "completed" 2022-03-08T23:07:30.335 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:07:30.339259 I | op-osd: creating OSD 3 on PVC "smithi135-lv-3-data-0d9nsm" 2022-03-08T23:07:30.336 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:07:30.339273 I | op-osd: OSD will have its main bluestore block on "smithi135-lv-3-data-0d9nsm" 2022-03-08T23:07:30.750 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:07:30.755624 I | op-osd: OSD orchestration status for PVC smithi135-lv-4-data-07k7pp is "completed" 2022-03-08T23:07:30.751 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:07:30.755660 I | op-osd: creating OSD 4 on PVC "smithi135-lv-4-data-07k7pp" 2022-03-08T23:07:30.751 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:07:30.755684 I | op-osd: OSD will have its main bluestore block on "smithi135-lv-4-data-07k7pp" 2022-03-08T23:07:32.437 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph osd stat -f json 2022-03-08T23:07:33.023 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:07:33.028461 I | op-osd: OSD orchestration status for node smithi106-lv-1-data-0j5llj is "orchestrating" 2022-03-08T23:07:33.547 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:07:33.552137 I | op-osd: OSD orchestration status for PVC smithi106-lv-1-data-0j5llj is "orchestrating" 2022-03-08T23:07:33.733 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-03-08T23:07:33.759 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":11,"num_osds":9,"num_up_osds":0,"osd_up_since":0,"num_in_osds":9,"osd_in_since":1646780852,"num_remapped_pgs":0} 2022-03-08T23:07:33.760 INFO:tasks.rook: have 0/12 OSDs 2022-03-08T23:07:34.401 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:07:34.405941 I | op-osd: OSD orchestration status for PVC smithi077-lv-4-data-0lbfgs is "completed" 2022-03-08T23:07:34.401 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:07:34.405962 I | op-osd: creating OSD 5 on PVC "smithi077-lv-4-data-0lbfgs" 2022-03-08T23:07:34.401 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:07:34.405969 I | op-osd: OSD will have its main bluestore block on "smithi077-lv-4-data-0lbfgs" 2022-03-08T23:07:36.693 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:07:36.698224 I | op-osd: OSD orchestration status for PVC smithi135-lv-2-data-0jcrvg is "completed" 2022-03-08T23:07:36.694 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:07:36.698256 I | op-osd: creating OSD 6 on PVC "smithi135-lv-2-data-0jcrvg" 2022-03-08T23:07:36.694 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:07:36.698273 I | op-osd: OSD will have its main bluestore block on "smithi135-lv-2-data-0jcrvg" 2022-03-08T23:07:38.605 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:07:38.609867 I | op-osd: OSD orchestration status for PVC smithi135-lv-1-data-0cq5dd is "completed" 2022-03-08T23:07:38.606 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:07:38.609913 I | op-osd: creating OSD 7 on PVC "smithi135-lv-1-data-0cq5dd" 2022-03-08T23:07:38.606 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:07:38.609939 I | op-osd: OSD will have its main bluestore block on "smithi135-lv-1-data-0cq5dd" 2022-03-08T23:07:39.822 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:07:39.825248 I | op-osd: OSD orchestration status for PVC smithi106-lv-3-data-07lcs6 is "completed" 2022-03-08T23:07:39.822 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:07:39.825273 I | op-osd: creating OSD 8 on PVC "smithi106-lv-3-data-07lcs6" 2022-03-08T23:07:39.822 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:07:39.825285 I | op-osd: OSD will have its main bluestore block on "smithi106-lv-3-data-07lcs6" 2022-03-08T23:07:41.195 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:07:41.199557 I | op-osd: OSD orchestration status for PVC smithi077-lv-1-data-0hb6kc is "completed" 2022-03-08T23:07:41.196 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:07:41.199579 I | op-osd: creating OSD 9 on PVC "smithi077-lv-1-data-0hb6kc" 2022-03-08T23:07:41.196 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:07:41.199586 I | op-osd: OSD will have its main bluestore block on "smithi077-lv-1-data-0hb6kc" 2022-03-08T23:07:43.530 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:07:43.533502 I | op-osd: OSD orchestration status for PVC smithi077-lv-2-data-0hqxxt is "completed" 2022-03-08T23:07:43.530 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:07:43.533532 I | op-osd: creating OSD 10 on PVC "smithi077-lv-2-data-0hqxxt" 2022-03-08T23:07:43.531 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:07:43.533542 I | op-osd: OSD will have its main bluestore block on "smithi077-lv-2-data-0hqxxt" 2022-03-08T23:07:43.760 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph osd stat -f json 2022-03-08T23:07:44.804 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-03-08T23:07:44.820 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":14,"num_osds":12,"num_up_osds":0,"osd_up_since":0,"num_in_osds":12,"osd_in_since":1646780858,"num_remapped_pgs":0} 2022-03-08T23:07:44.821 INFO:tasks.rook: have 0/12 OSDs 2022-03-08T23:07:45.723 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:07:45.725524 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". failed to reconcile cluster "rook-ceph": failed to configure local ceph cluster: failed to create cluster: failed to start ceph osds: 1 failures encountered while running osds on nodes in namespace "rook-ceph". 2022-03-08T23:07:45.723 INFO:tasks.rook.operator.smithi077.stdout:failed to provision OSD(s) on PVC smithi106-lv-1-data-0j5llj. &{OSDs:[] Status:failed PvcBackedOSD:true Message:failed to configure devices: failed to initialize devices on PVC: failed to run ceph-volume. Running command: /usr/bin/ceph-authtool --gen-print-key 2022-03-08T23:07:45.725 INFO:tasks.rook.operator.smithi077.stdout:Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 674c3b4d-af5e-4d6b-bb8a-fd0006b214b7 2022-03-08T23:07:45.725 INFO:tasks.rook.operator.smithi077.stdout: stderr: unable to get monitor info from DNS SRV with service name: ceph-mon 2022-03-08T23:07:45.725 INFO:tasks.rook.operator.smithi077.stdout: stderr: 2022-03-08T23:07:29.172+0000 7f66983d5700 -1 failed for service _ceph-mon._tcp 2022-03-08T23:07:45.726 INFO:tasks.rook.operator.smithi077.stdout: stderr: 2022-03-08T23:07:29.172+0000 7f66983d5700 -1 monclient: get_monmap_and_config cannot identify monitors to contact 2022-03-08T23:07:45.726 INFO:tasks.rook.operator.smithi077.stdout: stderr: [errno 2] RADOS object not found (error connecting to the cluster) 2022-03-08T23:07:45.726 INFO:tasks.rook.operator.smithi077.stdout:Traceback (most recent call last): 2022-03-08T23:07:45.726 INFO:tasks.rook.operator.smithi077.stdout: File "/usr/sbin/ceph-volume", line 11, in 2022-03-08T23:07:45.727 INFO:tasks.rook.operator.smithi077.stdout: load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2022-03-08T23:07:45.727 INFO:tasks.rook.operator.smithi077.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2022-03-08T23:07:45.727 INFO:tasks.rook.operator.smithi077.stdout: self.main(self.argv) 2022-03-08T23:07:45.727 INFO:tasks.rook.operator.smithi077.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2022-03-08T23:07:45.728 INFO:tasks.rook.operator.smithi077.stdout: return f(*a, **kw) 2022-03-08T23:07:45.728 INFO:tasks.rook.operator.smithi077.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2022-03-08T23:07:45.728 INFO:tasks.rook.operator.smithi077.stdout: terminal.dispatch(self.mapper, subcommand_args) 2022-03-08T23:07:45.728 INFO:tasks.rook.operator.smithi077.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2022-03-08T23:07:45.733 INFO:tasks.rook.operator.smithi077.stdout: instance.main() 2022-03-08T23:07:45.734 INFO:tasks.rook.operator.smithi077.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/main.py", line 32, in main 2022-03-08T23:07:45.734 INFO:tasks.rook.operator.smithi077.stdout: terminal.dispatch(self.mapper, self.argv) 2022-03-08T23:07:45.734 INFO:tasks.rook.operator.smithi077.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2022-03-08T23:07:45.734 INFO:tasks.rook.operator.smithi077.stdout: instance.main() 2022-03-08T23:07:45.734 INFO:tasks.rook.operator.smithi077.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 169, in main 2022-03-08T23:07:45.735 INFO:tasks.rook.operator.smithi077.stdout: self.safe_prepare(self.args) 2022-03-08T23:07:45.735 INFO:tasks.rook.operator.smithi077.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 91, in safe_prepare 2022-03-08T23:07:45.735 INFO:tasks.rook.operator.smithi077.stdout: self.prepare() 2022-03-08T23:07:45.735 INFO:tasks.rook.operator.smithi077.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2022-03-08T23:07:45.736 INFO:tasks.rook.operator.smithi077.stdout: return func(*a, **kw) 2022-03-08T23:07:45.736 INFO:tasks.rook.operator.smithi077.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 125, in prepare 2022-03-08T23:07:45.736 INFO:tasks.rook.operator.smithi077.stdout: osd_fsid, json.dumps(secrets)) 2022-03-08T23:07:45.736 INFO:tasks.rook.operator.smithi077.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/util/prepare.py", line 173, in create_id 2022-03-08T23:07:45.737 INFO:tasks.rook.operator.smithi077.stdout: raise RuntimeError('Unable to create a new OSD id') 2022-03-08T23:07:45.737 INFO:tasks.rook.operator.smithi077.stdout:RuntimeError: Unable to create a new OSD id. debug logs below: 2022-03-08T23:07:45.737 INFO:tasks.rook.operator.smithi077.stdout:[2022-03-08 23:07:28,584][ceph_volume.main][INFO ] Running command: ceph-volume --log-path /var/log/ceph/smithi106-lv-1-data-0j5llj raw prepare --bluestore --data /dev/mapper/vg_nvme-lv_1 2022-03-08T23:07:45.737 INFO:tasks.rook.operator.smithi077.stdout:[2022-03-08 23:07:28,585][ceph_volume.process][INFO ] Running command: /usr/bin/lsblk -plno KNAME,NAME,TYPE 2022-03-08T23:07:45.737 INFO:tasks.rook.operator.smithi077.stdout:[2022-03-08 23:07:28,592][ceph_volume.process][INFO ] stdout /dev/loop0 /dev/loop0 loop 2022-03-08T23:07:45.738 INFO:tasks.rook.operator.smithi077.stdout:[2022-03-08 23:07:28,592][ceph_volume.process][INFO ] stdout /dev/loop1 /dev/loop1 loop 2022-03-08T23:07:45.738 INFO:tasks.rook.operator.smithi077.stdout:[2022-03-08 23:07:28,592][ceph_volume.process][INFO ] stdout /dev/loop2 /dev/loop2 loop 2022-03-08T23:07:45.738 INFO:tasks.rook.operator.smithi077.stdout:[2022-03-08 23:07:28,592][ceph_volume.process][INFO ] stdout /dev/loop3 /dev/loop3 loop 2022-03-08T23:07:45.738 INFO:tasks.rook.operator.smithi077.stdout:[2022-03-08 23:07:28,593][ceph_volume.process][INFO ] stdout /dev/sda /dev/sda disk 2022-03-08T23:07:45.739 INFO:tasks.rook.operator.smithi077.stdout:[2022-03-08 23:07:28,593][ceph_volume.process][INFO ] stdout /dev/sda1 /dev/sda1 part 2022-03-08T23:07:45.739 INFO:tasks.rook.operator.smithi077.stdout:[2022-03-08 23:07:28,593][ceph_volume.process][INFO ] stdout /dev/dm-0 /dev/mapper/vg_nvme-lv_1 lvm 2022-03-08T23:07:45.739 INFO:tasks.rook.operator.smithi077.stdout:[2022-03-08 23:07:28,593][ceph_volume.process][INFO ] stdout /dev/dm-1 /dev/mapper/vg_nvme-lv_2 lvm 2022-03-08T23:07:45.739 INFO:tasks.rook.operator.smithi077.stdout:[2022-03-08 23:07:28,593][ceph_volume.process][INFO ] stdout /dev/dm-2 /dev/mapper/vg_nvme-lv_3 lvm 2022-03-08T23:07:45.740 INFO:tasks.rook.operator.smithi077.stdout:[2022-03-08 23:07:28,593][ceph_volume.process][INFO ] stdout /dev/dm-3 /dev/mapper/vg_nvme-lv_4 lvm 2022-03-08T23:07:45.740 INFO:tasks.rook.operator.smithi077.stdout:[2022-03-08 23:07:28,593][ceph_volume.process][INFO ] stdout /dev/dm-4 /dev/mapper/vg_nvme-lv_5 lvm 2022-03-08T23:07:45.740 INFO:tasks.rook.operator.smithi077.stdout:[2022-03-08 23:07:28,593][ceph_volume.process][INFO ] stdout /dev/nvme0n1 /dev/nvme0n1 disk 2022-03-08T23:07:45.740 INFO:tasks.rook.operator.smithi077.stdout:[2022-03-08 23:07:28,593][ceph_volume.process][INFO ] stdout /dev/nvme1n1 /dev/nvme1n1 disk 2022-03-08T23:07:45.740 INFO:tasks.rook.operator.smithi077.stdout:[2022-03-08 23:07:28,593][ceph_volume.process][INFO ] stdout /dev/nvme2n1 /dev/nvme2n1 disk 2022-03-08T23:07:45.741 INFO:tasks.rook.operator.smithi077.stdout:[2022-03-08 23:07:28,593][ceph_volume.process][INFO ] stdout /dev/nvme3n1 /dev/nvme3n1 disk 2022-03-08T23:07:45.741 INFO:tasks.rook.operator.smithi077.stdout:[2022-03-08 23:07:28,593][ceph_volume.process][INFO ] stdout /dev/nvme4n1 /dev/nvme4n1 disk 2022-03-08T23:07:45.741 INFO:tasks.rook.operator.smithi077.stdout:[2022-03-08 23:07:28,601][ceph_volume.util.system][WARNING] Executable lvs not found on the host, will return lvs as-is 2022-03-08T23:07:45.741 INFO:tasks.rook.operator.smithi077.stdout:[2022-03-08 23:07:28,601][ceph_volume.process][INFO ] Running command: lvs --noheadings --readonly --separator=";" -a --units=b --nosuffix -S lv_path=/dev/mapper/vg_nvme-lv_1 -o lv_tags,lv_path,lv_name,vg_name,lv_uuid,lv_size 2022-03-08T23:07:45.742 INFO:tasks.rook.operator.smithi077.stdout:[2022-03-08 23:07:29,021][ceph_volume.process][INFO ] Running command: /usr/bin/lsblk --nodeps -P -o NAME,KNAME,MAJ:MIN,FSTYPE,MOUNTPOINT,LABEL,UUID,RO,RM,MODEL,SIZE,STATE,OWNER,GROUP,MODE,ALIGNMENT,PHY-SEC,LOG-SEC,ROTA,SCHED,TYPE,DISC-ALN,DISC-GRAN,DISC-MAX,DISC-ZERO,PKNAME,PARTLABEL /dev/mapper/vg_nvme-lv_1 2022-03-08T23:07:45.742 INFO:tasks.rook.operator.smithi077.stdout:[2022-03-08 23:07:29,029][ceph_volume.process][INFO ] stdout NAME="vg_nvme-lv_1" KNAME="dm-0" MAJ:MIN="253:0" FSTYPE="" MOUNTPOINT="" LABEL="" UUID="" RO="0" RM="0" MODEL="" SIZE="89.4G" STATE="running" OWNER="root" GROUP="disk" MODE="brw-rw----" ALIGNMENT="0" PHY-SEC="512" LOG-SEC="512" ROTA="0" SCHED="" TYPE="lvm" DISC-ALN="0" DISC-GRAN="512B" DISC-MAX="2T" DISC-ZERO="0" PKNAME="" PARTLABEL="" 2022-03-08T23:07:45.742 INFO:tasks.rook.operator.smithi077.stdout:[2022-03-08 23:07:29,029][ceph_volume.process][INFO ] Running command: /usr/sbin/blkid -c /dev/null -p /dev/mapper/vg_nvme-lv_1 2022-03-08T23:07:45.742 INFO:tasks.rook.operator.smithi077.stdout:[2022-03-08 23:07:29,032][ceph_volume.util.disk][INFO ] opening device /dev/mapper/vg_nvme-lv_1 to check for BlueStore label 2022-03-08T23:07:45.742 INFO:tasks.rook.operator.smithi077.stdout:[2022-03-08 23:07:29,032][ceph_volume.util.disk][INFO ] opening device /dev/mapper/vg_nvme-lv_1 to check for BlueStore label 2022-03-08T23:07:45.743 INFO:tasks.rook.operator.smithi077.stdout:[2022-03-08 23:07:29,032][ceph_volume.process][INFO ] Running command: /usr/sbin/udevadm info --query=property /dev/mapper/vg_nvme-lv_1 2022-03-08T23:07:45.743 INFO:tasks.rook.operator.smithi077.stdout:[2022-03-08 23:07:29,045][ceph_volume.process][INFO ] stdout DEVLINKS=/dev/vg_nvme/lv_1 /dev/disk/by-id/dm-uuid-LVM-tnFwFsOQYo0FTayanllK4BxIyJ5MyZU17n1F81A27eTqUDSQa4J8NKnLim07bvel /dev/mapper/vg_nvme-lv_1 /dev/disk/by-id/dm-name-vg_nvme-lv_1 2022-03-08T23:07:45.743 INFO:tasks.rook.operator.smithi077.stdout:[2022-03-08 23:07:29,045][ceph_volume.process][INFO ] stdout DEVNAME=/dev/dm-0 2022-03-08T23:07:45.743 INFO:tasks.rook.operator.smithi077.stdout:[2022-03-08 23:07:29,045][ceph_volume.process][INFO ] stdout DEVPATH=/devices/virtual/block/dm-0 2022-03-08T23:07:45.744 INFO:tasks.rook.operator.smithi077.stdout:[2022-03-08 23:07:29,045][ceph_volume.process][INFO ] stdout DEVTYPE=disk 2022-03-08T23:07:45.744 INFO:tasks.rook.operator.smithi077.stdout:[2022-03-08 23:07:29,046][ceph_volume.process][INFO ] stdout DM_LV_NAME=lv_1 2022-03-08T23:07:45.744 INFO:tasks.rook.operator.smithi077.stdout:[2022-03-08 23:07:29,046][ceph_volume.process][INFO ] stdout DM_NAME=vg_nvme-lv_1 2022-03-08T23:07:45.744 INFO:tasks.rook.operator.smithi077.stdout:[2022-03-08 23:07:29,046][ceph_volume.process][INFO ] stdout DM_SUSPENDED=0 2022-03-08T23:07:45.745 INFO:tasks.rook.operator.smithi077.stdout:[2022-03-08 23:07:29,046][ceph_volume.process][INFO ] stdout DM_UDEV_DISABLE_LIBRARY_FALLBACK_FLAG=1 2022-03-08T23:07:45.745 INFO:tasks.rook.operator.smithi077.stdout:[2022-03-08 23:07:29,046][ceph_volume.process][INFO ] stdout DM_UDEV_PRIMARY_SOURCE_FLAG=1 2022-03-08T23:07:45.745 INFO:tasks.rook.operator.smithi077.stdout:[2022-03-08 23:07:29,046][ceph_volume.process][INFO ] stdout DM_UDEV_RULES=1 2022-03-08T23:07:45.745 INFO:tasks.rook.operator.smithi077.stdout:[2022-03-08 23:07:29,046][ceph_volume.process][INFO ] stdout DM_UDEV_RULES_VSN=2 2022-03-08T23:07:45.745 INFO:tasks.rook.operator.smithi077.stdout:[2022-03-08 23:07:29,046][ceph_volume.process][INFO ] stdout DM_UUID=LVM-tnFwFsOQYo0FTayanllK4BxIyJ5MyZU17n1F81A27eTqUDSQa4J8NKnLim07bvel 2022-03-08T23:07:45.746 INFO:tasks.rook.operator.smithi077.stdout:[2022-03-08 23:07:29,046][ceph_volume.process][INFO ] stdout DM_VG_NAME=vg_nvme 2022-03-08T23:07:45.746 INFO:tasks.rook.operator.smithi077.stdout:[2022-03-08 23:07:29,046][ceph_volume.process][INFO ] stdout MAJOR=253 2022-03-08T23:07:45.746 INFO:tasks.rook.operator.smithi077.stdout:[2022-03-08 23:07:29,046][ceph_volume.process][INFO ] stdout MINOR=0 2022-03-08T23:07:45.746 INFO:tasks.rook.operator.smithi077.stdout:[2022-03-08 23:07:29,046][ceph_volume.process][INFO ] stdout SUBSYSTEM=block 2022-03-08T23:07:45.747 INFO:tasks.rook.operator.smithi077.stdout:[2022-03-08 23:07:29,046][ceph_volume.process][INFO ] stdout TAGS=:systemd: 2022-03-08T23:07:45.747 INFO:tasks.rook.operator.smithi077.stdout:[2022-03-08 23:07:29,046][ceph_volume.process][INFO ] stdout USEC_INITIALIZED=197899962 2022-03-08T23:07:45.747 INFO:tasks.rook.operator.smithi077.stdout:[2022-03-08 23:07:29,046][ceph_volume.process][INFO ] Running command: /usr/bin/ceph-authtool --gen-print-key 2022-03-08T23:07:45.747 INFO:tasks.rook.operator.smithi077.stdout:[2022-03-08 23:07:29,064][ceph_volume.process][INFO ] stdout AQCx4Sdi1/fDAxAAOXt0E8Z5e1UmxUpX//gVvA== 2022-03-08T23:07:45.748 INFO:tasks.rook.operator.smithi077.stdout:[2022-03-08 23:07:29,068][ceph_volume.process][INFO ] Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 674c3b4d-af5e-4d6b-bb8a-fd0006b214b7 2022-03-08T23:07:45.748 INFO:tasks.rook.operator.smithi077.stdout:[2022-03-08 23:07:29,181][ceph_volume.process][INFO ] stderr unable to get monitor info from DNS SRV with service name: ceph-mon 2022-03-08T23:07:45.751 INFO:tasks.rook.operator.smithi077.stdout:[2022-03-08 23:07:29,181][ceph_volume.process][INFO ] stderr 2022-03-08T23:07:29.172+0000 7f66983d5700 -1 failed for service _ceph-mon._tcp 2022-03-08T23:07:45.751 INFO:tasks.rook.operator.smithi077.stdout:[2022-03-08 23:07:29,181][ceph_volume.process][INFO ] stderr 2022-03-08T23:07:29.172+0000 7f66983d5700 -1 monclient: get_monmap_and_config cannot identify monitors to contact 2022-03-08T23:07:45.752 INFO:tasks.rook.operator.smithi077.stdout:[2022-03-08 23:07:29,181][ceph_volume.process][INFO ] stderr [errno 2] RADOS object not found (error connecting to the cluster) 2022-03-08T23:07:45.752 INFO:tasks.rook.operator.smithi077.stdout:[2022-03-08 23:07:29,181][ceph_volume.devices.raw.prepare][ERROR ] raw prepare was unable to complete 2022-03-08T23:07:45.752 INFO:tasks.rook.operator.smithi077.stdout:Traceback (most recent call last): 2022-03-08T23:07:45.752 INFO:tasks.rook.operator.smithi077.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 91, in safe_prepare 2022-03-08T23:07:45.752 INFO:tasks.rook.operator.smithi077.stdout: self.prepare() 2022-03-08T23:07:45.753 INFO:tasks.rook.operator.smithi077.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2022-03-08T23:07:45.753 INFO:tasks.rook.operator.smithi077.stdout: return func(*a, **kw) 2022-03-08T23:07:45.753 INFO:tasks.rook.operator.smithi077.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 125, in prepare 2022-03-08T23:07:45.753 INFO:tasks.rook.operator.smithi077.stdout: osd_fsid, json.dumps(secrets)) 2022-03-08T23:07:45.754 INFO:tasks.rook.operator.smithi077.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/util/prepare.py", line 173, in create_id 2022-03-08T23:07:45.754 INFO:tasks.rook.operator.smithi077.stdout: raise RuntimeError('Unable to create a new OSD id') 2022-03-08T23:07:45.754 INFO:tasks.rook.operator.smithi077.stdout:RuntimeError: Unable to create a new OSD id 2022-03-08T23:07:45.754 INFO:tasks.rook.operator.smithi077.stdout:[2022-03-08 23:07:29,183][ceph_volume.devices.raw.prepare][INFO ] will rollback OSD ID creation 2022-03-08T23:07:45.754 INFO:tasks.rook.operator.smithi077.stdout:[2022-03-08 23:07:29,183][ceph_volume][ERROR ] exception caught by decorator 2022-03-08T23:07:45.755 INFO:tasks.rook.operator.smithi077.stdout:Traceback (most recent call last): 2022-03-08T23:07:45.755 INFO:tasks.rook.operator.smithi077.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2022-03-08T23:07:45.755 INFO:tasks.rook.operator.smithi077.stdout: return f(*a, **kw) 2022-03-08T23:07:45.755 INFO:tasks.rook.operator.smithi077.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2022-03-08T23:07:45.756 INFO:tasks.rook.operator.smithi077.stdout: terminal.dispatch(self.mapper, subcommand_args) 2022-03-08T23:07:45.756 INFO:tasks.rook.operator.smithi077.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2022-03-08T23:07:45.756 INFO:tasks.rook.operator.smithi077.stdout: instance.main() 2022-03-08T23:07:45.756 INFO:tasks.rook.operator.smithi077.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/main.py", line 32, in main 2022-03-08T23:07:45.756 INFO:tasks.rook.operator.smithi077.stdout: terminal.dispatch(self.mapper, self.argv) 2022-03-08T23:07:45.757 INFO:tasks.rook.operator.smithi077.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2022-03-08T23:07:45.757 INFO:tasks.rook.operator.smithi077.stdout: instance.main() 2022-03-08T23:07:45.757 INFO:tasks.rook.operator.smithi077.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 169, in main 2022-03-08T23:07:45.757 INFO:tasks.rook.operator.smithi077.stdout: self.safe_prepare(self.args) 2022-03-08T23:07:45.758 INFO:tasks.rook.operator.smithi077.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 91, in safe_prepare 2022-03-08T23:07:45.758 INFO:tasks.rook.operator.smithi077.stdout: self.prepare() 2022-03-08T23:07:45.758 INFO:tasks.rook.operator.smithi077.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2022-03-08T23:07:45.758 INFO:tasks.rook.operator.smithi077.stdout: return func(*a, **kw) 2022-03-08T23:07:45.759 INFO:tasks.rook.operator.smithi077.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 125, in prepare 2022-03-08T23:07:45.759 INFO:tasks.rook.operator.smithi077.stdout: osd_fsid, json.dumps(secrets)) 2022-03-08T23:07:45.759 INFO:tasks.rook.operator.smithi077.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/util/prepare.py", line 173, in create_id 2022-03-08T23:07:45.759 INFO:tasks.rook.operator.smithi077.stdout: raise RuntimeError('Unable to create a new OSD id') 2022-03-08T23:07:45.759 INFO:tasks.rook.operator.smithi077.stdout:RuntimeError: Unable to create a new OSD id 2022-03-08T23:07:45.760 INFO:tasks.rook.operator.smithi077.stdout:: exit status 1} 2022-03-08T23:07:45.760 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:07:45.731296 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-03-08T23:07:46.474 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:07:46.476988 I | op-mon: parsing mon endpoints: a=10.110.141.123:6789,b=10.98.41.230:6789,c=10.109.47.142:6789 2022-03-08T23:07:46.957 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:07:46.961212 I | ceph-spec: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3... 2022-03-08T23:07:54.822 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph osd stat -f json 2022-03-08T23:07:56.803 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-03-08T23:07:56.826 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":15,"num_osds":12,"num_up_osds":0,"osd_up_since":0,"num_in_osds":12,"osd_in_since":1646780858,"num_remapped_pgs":0} 2022-03-08T23:07:56.826 INFO:tasks.rook: have 0/12 OSDs 2022-03-08T23:08:01.691 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:01.693732 I | ceph-spec: detected ceph image version: "17.0.0-11025 quincy" 2022-03-08T23:08:01.692 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:01.693769 I | ceph-cluster-controller: validating ceph version from provided image 2022-03-08T23:08:01.692 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:01.693788 W | ceph-cluster-controller: unsupported ceph version detected: "17.0.0-11025 quincy", pursuing 2022-03-08T23:08:01.966 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:01.960084 I | op-mon: parsing mon endpoints: a=10.110.141.123:6789,b=10.98.41.230:6789,c=10.109.47.142:6789 2022-03-08T23:08:01.984 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:01.985206 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-08T23:08:01.984 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:01.985382 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-08T23:08:03.666 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:03.667902 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-11025 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3" 2022-03-08T23:08:05.086 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:05.088069 I | op-mon: start running mons 2022-03-08T23:08:05.111 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:05.112789 I | op-mon: parsing mon endpoints: a=10.110.141.123:6789,b=10.98.41.230:6789,c=10.109.47.142:6789 2022-03-08T23:08:05.648 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:05.650192 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.110.141.123:6789","10.98.41.230:6789","10.109.47.142:6789"]}] data:a=10.110.141.123:6789,b=10.98.41.230:6789,c=10.109.47.142:6789 mapping:{"node":{"a":{"Name":"smithi135","Hostname":"smithi135","Address":"172.21.15.135"},"b":{"Name":"smithi106","Hostname":"smithi106","Address":"172.21.15.106"},"c":{"Name":"smithi077","Hostname":"smithi077","Address":"172.21.15.77"}}} maxMonId:2] 2022-03-08T23:08:05.654 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:05.656329 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-08T23:08:05.656 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:05.656483 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-08T23:08:06.690 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:06.691929 I | op-mon: targeting the mon count 3 2022-03-08T23:08:06.695 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:06.697284 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-08T23:08:06.827 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph osd stat -f json 2022-03-08T23:08:07.275 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:07.276685 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-08T23:08:07.275 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:07.276747 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-03-08T23:08:08.279 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-03-08T23:08:08.302 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":21,"num_osds":12,"num_up_osds":5,"osd_up_since":1646780886,"num_in_osds":12,"osd_in_since":1646780858,"num_remapped_pgs":0} 2022-03-08T23:08:08.307 INFO:tasks.rook: have 5/12 OSDs 2022-03-08T23:08:08.564 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:08.565566 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-03-08T23:08:08.564 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:08.565590 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-08T23:08:09.382 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:09.383593 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-08T23:08:09.382 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:09.383621 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-03-08T23:08:09.718 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:09.719368 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-03-08T23:08:09.718 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:09.719393 I | op-config: deleting "log file" option from the mon configuration database 2022-03-08T23:08:10.057 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:10.058377 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-03-08T23:08:10.057 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:10.058399 I | op-mon: checking for basic quorum with existing mons 2022-03-08T23:08:11.426 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:11.427837 I | op-mon: mon "c" endpoint is [v2:10.109.47.142:3300,v1:10.109.47.142:6789] 2022-03-08T23:08:13.088 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:13.089489 I | op-mon: mon "a" endpoint is [v2:10.110.141.123:3300,v1:10.110.141.123:6789] 2022-03-08T23:08:13.359 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:13.360488 I | op-mon: mon "b" endpoint is [v2:10.98.41.230:3300,v1:10.98.41.230:6789] 2022-03-08T23:08:13.454 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:13.455357 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.110.141.123:6789","10.98.41.230:6789","10.109.47.142:6789"]}] data:a=10.110.141.123:6789,b=10.98.41.230:6789,c=10.109.47.142:6789 mapping:{"node":{"a":{"Name":"smithi135","Hostname":"smithi135","Address":"172.21.15.135"},"b":{"Name":"smithi106","Hostname":"smithi106","Address":"172.21.15.106"},"c":{"Name":"smithi077","Hostname":"smithi077","Address":"172.21.15.77"}}} maxMonId:2] 2022-03-08T23:08:13.460 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:13.462613 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-08T23:08:13.464 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:13.462855 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-08T23:08:13.519 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:13.520575 I | op-mon: deployment for mon rook-ceph-mon-c already exists. updating if needed 2022-03-08T23:08:13.530 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:13.532469 I | op-k8sutil: deployment "rook-ceph-mon-c" did not change, nothing to update 2022-03-08T23:08:13.531 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:13.532494 I | op-mon: waiting for mon quorum with [c a b] 2022-03-08T23:08:14.127 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:14.129293 I | op-mon: mons running: [c a b] 2022-03-08T23:08:14.598 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:14.599441 I | op-mon: Monitors in quorum: [a b c] 2022-03-08T23:08:14.604 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:14.606337 I | op-mon: deployment for mon rook-ceph-mon-a already exists. updating if needed 2022-03-08T23:08:14.613 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:14.615070 I | op-k8sutil: deployment "rook-ceph-mon-a" did not change, nothing to update 2022-03-08T23:08:14.613 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:14.615095 I | op-mon: waiting for mon quorum with [c a b] 2022-03-08T23:08:14.701 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:14.702490 I | op-mon: mons running: [c a b] 2022-03-08T23:08:15.107 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:15.108560 I | op-mon: Monitors in quorum: [a b c] 2022-03-08T23:08:15.115 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:15.116780 I | op-mon: deployment for mon rook-ceph-mon-b already exists. updating if needed 2022-03-08T23:08:15.124 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:15.125931 I | op-k8sutil: deployment "rook-ceph-mon-b" did not change, nothing to update 2022-03-08T23:08:15.124 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:15.125955 I | op-mon: waiting for mon quorum with [c a b] 2022-03-08T23:08:15.336 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:15.337756 I | op-mon: mons running: [c a b] 2022-03-08T23:08:15.755 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:15.756485 I | op-mon: Monitors in quorum: [a b c] 2022-03-08T23:08:15.755 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:15.756518 I | op-mon: mons created: 3 2022-03-08T23:08:16.179 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:16.180269 I | op-mon: waiting for mon quorum with [c a b] 2022-03-08T23:08:16.223 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:16.224624 I | op-mon: mons running: [c a b] 2022-03-08T23:08:16.649 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:16.650647 I | op-mon: Monitors in quorum: [a b c] 2022-03-08T23:08:16.650 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:16.650692 I | cephclient: getting or creating ceph auth key "client.csi-rbd-provisioner" 2022-03-08T23:08:17.056 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:17.057360 I | cephclient: getting or creating ceph auth key "client.csi-rbd-node" 2022-03-08T23:08:17.475 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:17.476581 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-provisioner" 2022-03-08T23:08:17.945 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:17.946754 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-node" 2022-03-08T23:08:18.309 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph osd stat -f json 2022-03-08T23:08:18.397 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:18.398093 I | ceph-csi: created kubernetes csi secrets for cluster "rook-ceph" 2022-03-08T23:08:18.397 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:18.398115 I | cephclient: getting or creating ceph auth key "client.crash" 2022-03-08T23:08:18.946 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-03-08T23:08:18.972 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":30,"num_osds":12,"num_up_osds":10,"osd_up_since":1646780898,"num_in_osds":12,"osd_in_since":1646780858,"num_remapped_pgs":0} 2022-03-08T23:08:18.973 INFO:tasks.rook: have 10/12 OSDs 2022-03-08T23:08:18.974 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:18.975493 I | ceph-crashcollector-controller: created kubernetes crash collector secret for cluster "rook-ceph" 2022-03-08T23:08:19.372 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:19.372749 I | cephclient: successfully enabled msgr2 protocol 2022-03-08T23:08:19.372 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:19.372792 I | cephclient: create rbd-mirror bootstrap peer token "client.rbd-mirror-peer" 2022-03-08T23:08:19.372 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:19.372804 I | cephclient: getting or creating ceph auth key "client.rbd-mirror-peer" 2022-03-08T23:08:19.799 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:19.800472 I | cephclient: successfully created rbd-mirror bootstrap peer token for cluster "rook-ceph" 2022-03-08T23:08:19.828 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:19.829467 I | op-mgr: start running mgr 2022-03-08T23:08:19.828 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:19.829507 I | cephclient: getting or creating ceph auth key "mgr.a" 2022-03-08T23:08:20.349 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:20.350651 I | op-mgr: deployment for mgr rook-ceph-mgr-a already exists. updating if needed 2022-03-08T23:08:20.358 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:20.360071 I | op-k8sutil: deployment "rook-ceph-mgr-a" did not change, nothing to update 2022-03-08T23:08:20.362 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:20.363469 I | op-mgr: setting services to point to mgr "a" 2022-03-08T23:08:20.436 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:20.437327 I | op-mgr: no need to update service "rook-ceph-mgr" 2022-03-08T23:08:20.436 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:20.437493 I | op-mgr: successful modules: balancer 2022-03-08T23:08:20.447 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:20.448557 I | op-osd: start running osds in namespace "rook-ceph" 2022-03-08T23:08:20.447 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:20.448575 I | op-osd: wait timeout for healthy OSDs during upgrade or restart is "10m0s" 2022-03-08T23:08:20.553 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:20.554052 I | op-osd: start provisioning the OSDs on PVCs, if needed 2022-03-08T23:08:20.559 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:20.560339 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi077-lv-1" 2022-03-08T23:08:20.559 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:20.560394 I | op-osd: OSD PVC "smithi077-lv-1-data-0hb6kc" already exists 2022-03-08T23:08:20.560 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:20.560410 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi077-lv-2" 2022-03-08T23:08:20.560 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:20.560424 I | op-osd: OSD PVC "smithi077-lv-2-data-0hqxxt" already exists 2022-03-08T23:08:20.560 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:20.560437 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi077-lv-3" 2022-03-08T23:08:20.560 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:20.560449 I | op-osd: OSD PVC "smithi077-lv-3-data-0zvk2m" already exists 2022-03-08T23:08:20.563 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:20.560459 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi077-lv-4" 2022-03-08T23:08:20.563 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:20.560472 I | op-osd: OSD PVC "smithi077-lv-4-data-0lbfgs" already exists 2022-03-08T23:08:20.564 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:20.560481 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi106-lv-1" 2022-03-08T23:08:20.564 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:20.560493 I | op-osd: OSD PVC "smithi106-lv-1-data-0j5llj" already exists 2022-03-08T23:08:20.564 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:20.560502 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi106-lv-2" 2022-03-08T23:08:20.564 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:20.560514 I | op-osd: OSD PVC "smithi106-lv-2-data-0r5dlw" already exists 2022-03-08T23:08:20.565 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:20.560522 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi106-lv-3" 2022-03-08T23:08:20.565 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:20.560533 I | op-osd: OSD PVC "smithi106-lv-3-data-07lcs6" already exists 2022-03-08T23:08:20.565 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:20.560541 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi106-lv-4" 2022-03-08T23:08:20.565 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:20.560552 I | op-osd: OSD PVC "smithi106-lv-4-data-04hgzm" already exists 2022-03-08T23:08:20.566 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:20.560560 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi135-lv-1" 2022-03-08T23:08:20.566 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:20.560571 I | op-osd: OSD PVC "smithi135-lv-1-data-0cq5dd" already exists 2022-03-08T23:08:20.567 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:20.560582 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi135-lv-2" 2022-03-08T23:08:20.567 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:20.560594 I | op-osd: OSD PVC "smithi135-lv-2-data-0jcrvg" already exists 2022-03-08T23:08:20.567 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:20.560603 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi135-lv-3" 2022-03-08T23:08:20.567 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:20.560614 I | op-osd: OSD PVC "smithi135-lv-3-data-0d9nsm" already exists 2022-03-08T23:08:20.567 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:20.560622 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi135-lv-4" 2022-03-08T23:08:20.568 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:20.560635 I | op-osd: OSD PVC "smithi135-lv-4-data-07k7pp" already exists 2022-03-08T23:08:20.586 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:20.586842 I | op-osd: OSD will have its main bluestore block on "smithi077-lv-1-data-0hb6kc" 2022-03-08T23:08:20.586 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:20.586866 I | op-osd: skipping OSD prepare job creation for PVC "smithi077-lv-1-data-0hb6kc" because OSD daemon using the PVC already exists 2022-03-08T23:08:20.587 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:20.586873 I | op-osd: OSD will have its main bluestore block on "smithi077-lv-2-data-0hqxxt" 2022-03-08T23:08:20.587 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:20.586878 I | op-osd: skipping OSD prepare job creation for PVC "smithi077-lv-2-data-0hqxxt" because OSD daemon using the PVC already exists 2022-03-08T23:08:20.587 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:20.586882 I | op-osd: OSD will have its main bluestore block on "smithi077-lv-3-data-0zvk2m" 2022-03-08T23:08:20.587 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:20.586887 I | op-osd: skipping OSD prepare job creation for PVC "smithi077-lv-3-data-0zvk2m" because OSD daemon using the PVC already exists 2022-03-08T23:08:20.588 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:20.586892 I | op-osd: OSD will have its main bluestore block on "smithi077-lv-4-data-0lbfgs" 2022-03-08T23:08:20.588 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:20.586897 I | op-osd: skipping OSD prepare job creation for PVC "smithi077-lv-4-data-0lbfgs" because OSD daemon using the PVC already exists 2022-03-08T23:08:20.588 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:20.586901 I | op-osd: OSD will have its main bluestore block on "smithi106-lv-1-data-0j5llj" 2022-03-08T23:08:21.002 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:21.003210 I | op-k8sutil: Removing previous job rook-ceph-osd-prepare-smithi106-lv-1-data-0j5llj to start a new one 2022-03-08T23:08:21.031 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:21.032705 I | op-k8sutil: batch job rook-ceph-osd-prepare-smithi106-lv-1-data-0j5llj still exists 2022-03-08T23:08:21.372 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:21.373300 I | op-mgr: successful modules: mgr module(s) from the spec 2022-03-08T23:08:21.376 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:21.377679 I | op-mgr: successful modules: prometheus 2022-03-08T23:08:21.379 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:21.378291 I | op-mgr: successful modules: dashboard 2022-03-08T23:08:23.163 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:23.163929 I | op-mgr: successful modules: orchestrator modules 2022-03-08T23:08:24.040 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:24.040781 I | op-k8sutil: batch job rook-ceph-osd-prepare-smithi106-lv-1-data-0j5llj deleted 2022-03-08T23:08:24.052 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:24.052389 I | op-osd: started OSD provisioning job for PVC "smithi106-lv-1-data-0j5llj" 2022-03-08T23:08:24.053 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:24.052428 I | op-osd: OSD will have its main bluestore block on "smithi106-lv-2-data-0r5dlw" 2022-03-08T23:08:24.053 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:24.052434 I | op-osd: skipping OSD prepare job creation for PVC "smithi106-lv-2-data-0r5dlw" because OSD daemon using the PVC already exists 2022-03-08T23:08:24.053 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:24.052438 I | op-osd: OSD will have its main bluestore block on "smithi106-lv-3-data-07lcs6" 2022-03-08T23:08:24.053 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:24.052442 I | op-osd: skipping OSD prepare job creation for PVC "smithi106-lv-3-data-07lcs6" because OSD daemon using the PVC already exists 2022-03-08T23:08:24.054 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:24.052448 I | op-osd: OSD will have its main bluestore block on "smithi106-lv-4-data-04hgzm" 2022-03-08T23:08:24.054 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:24.052452 I | op-osd: skipping OSD prepare job creation for PVC "smithi106-lv-4-data-04hgzm" because OSD daemon using the PVC already exists 2022-03-08T23:08:24.054 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:24.052455 I | op-osd: OSD will have its main bluestore block on "smithi135-lv-1-data-0cq5dd" 2022-03-08T23:08:24.054 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:24.052461 I | op-osd: skipping OSD prepare job creation for PVC "smithi135-lv-1-data-0cq5dd" because OSD daemon using the PVC already exists 2022-03-08T23:08:24.055 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:24.052466 I | op-osd: OSD will have its main bluestore block on "smithi135-lv-2-data-0jcrvg" 2022-03-08T23:08:24.055 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:24.052470 I | op-osd: skipping OSD prepare job creation for PVC "smithi135-lv-2-data-0jcrvg" because OSD daemon using the PVC already exists 2022-03-08T23:08:24.055 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:24.052475 I | op-osd: OSD will have its main bluestore block on "smithi135-lv-3-data-0d9nsm" 2022-03-08T23:08:24.055 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:24.052479 I | op-osd: skipping OSD prepare job creation for PVC "smithi135-lv-3-data-0d9nsm" because OSD daemon using the PVC already exists 2022-03-08T23:08:24.056 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:24.052483 I | op-osd: OSD will have its main bluestore block on "smithi135-lv-4-data-07k7pp" 2022-03-08T23:08:24.056 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:24.052488 I | op-osd: skipping OSD prepare job creation for PVC "smithi135-lv-4-data-07k7pp" because OSD daemon using the PVC already exists 2022-03-08T23:08:24.056 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:24.052492 I | op-osd: start provisioning the OSDs on nodes, if needed 2022-03-08T23:08:24.056 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:24.052496 I | op-osd: no nodes are defined for configuring OSDs on raw devices 2022-03-08T23:08:24.069 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:24.070861 I | op-osd: OSD orchestration status for PVC smithi106-lv-1-data-0j5llj is "starting" 2022-03-08T23:08:24.171 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:24.172342 I | op-osd: skipping osd checks for ok-to-stop 2022-03-08T23:08:24.180 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:24.181853 I | op-osd: updating OSD 0 on PVC "smithi077-lv-3-data-0zvk2m" 2022-03-08T23:08:24.181 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:24.181881 I | op-osd: OSD will have its main bluestore block on "smithi077-lv-3-data-0zvk2m" 2022-03-08T23:08:24.365 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:24.366362 I | op-osd: skipping osd checks for ok-to-stop 2022-03-08T23:08:24.400 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:24.401643 I | op-osd: updating OSD 1 on PVC "smithi106-lv-2-data-0r5dlw" 2022-03-08T23:08:24.401 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:24.401684 I | op-osd: OSD will have its main bluestore block on "smithi106-lv-2-data-0r5dlw" 2022-03-08T23:08:24.572 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:24.572921 I | op-osd: skipping osd checks for ok-to-stop 2022-03-08T23:08:24.577 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:24.578390 I | op-osd: updating OSD 10 on PVC "smithi077-lv-2-data-0hqxxt" 2022-03-08T23:08:24.577 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:24.578421 I | op-osd: OSD will have its main bluestore block on "smithi077-lv-2-data-0hqxxt" 2022-03-08T23:08:24.779 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:24.780695 I | op-osd: skipping osd checks for ok-to-stop 2022-03-08T23:08:24.786 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:24.787222 I | op-osd: updating OSD 2 on PVC "smithi106-lv-4-data-04hgzm" 2022-03-08T23:08:24.786 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:24.787256 I | op-osd: OSD will have its main bluestore block on "smithi106-lv-4-data-04hgzm" 2022-03-08T23:08:25.483 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:25.483847 I | op-osd: skipping osd checks for ok-to-stop 2022-03-08T23:08:25.578 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:25.579353 I | op-osd: updating OSD 3 on PVC "smithi135-lv-3-data-0d9nsm" 2022-03-08T23:08:25.579 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:25.579378 I | op-osd: OSD will have its main bluestore block on "smithi135-lv-3-data-0d9nsm" 2022-03-08T23:08:26.283 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:26.284455 I | op-osd: skipping osd checks for ok-to-stop 2022-03-08T23:08:26.402 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:26.402576 I | op-osd: updating OSD 4 on PVC "smithi135-lv-4-data-07k7pp" 2022-03-08T23:08:26.402 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:26.402621 I | op-osd: OSD will have its main bluestore block on "smithi135-lv-4-data-07k7pp" 2022-03-08T23:08:27.083 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:27.084444 I | op-osd: skipping osd checks for ok-to-stop 2022-03-08T23:08:27.183 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:27.182504 I | op-osd: updating OSD 5 on PVC "smithi077-lv-4-data-0lbfgs" 2022-03-08T23:08:27.183 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:27.182561 I | op-osd: OSD will have its main bluestore block on "smithi077-lv-4-data-0lbfgs" 2022-03-08T23:08:27.887 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:27.887517 I | op-osd: skipping osd checks for ok-to-stop 2022-03-08T23:08:27.977 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:27.978515 I | op-osd: updating OSD 6 on PVC "smithi135-lv-2-data-0jcrvg" 2022-03-08T23:08:27.978 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:27.978551 I | op-osd: OSD will have its main bluestore block on "smithi135-lv-2-data-0jcrvg" 2022-03-08T23:08:28.685 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:28.686029 I | op-osd: skipping osd checks for ok-to-stop 2022-03-08T23:08:28.777 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:28.778647 I | op-osd: updating OSD 7 on PVC "smithi135-lv-1-data-0cq5dd" 2022-03-08T23:08:28.777 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:28.778671 I | op-osd: OSD will have its main bluestore block on "smithi135-lv-1-data-0cq5dd" 2022-03-08T23:08:28.974 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph osd stat -f json 2022-03-08T23:08:29.484 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:29.484556 I | op-osd: skipping osd checks for ok-to-stop 2022-03-08T23:08:29.494 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-03-08T23:08:29.512 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":32,"num_osds":12,"num_up_osds":11,"osd_up_since":1646780899,"num_in_osds":12,"osd_in_since":1646780858,"num_remapped_pgs":0} 2022-03-08T23:08:29.513 INFO:tasks.rook: have 11/12 OSDs 2022-03-08T23:08:29.599 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:29.599564 I | op-osd: updating OSD 8 on PVC "smithi106-lv-3-data-07lcs6" 2022-03-08T23:08:29.599 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:29.599594 I | op-osd: OSD will have its main bluestore block on "smithi106-lv-3-data-07lcs6" 2022-03-08T23:08:30.281 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:30.282330 I | op-osd: skipping osd checks for ok-to-stop 2022-03-08T23:08:30.377 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:30.378817 I | op-osd: updating OSD 9 on PVC "smithi077-lv-1-data-0hb6kc" 2022-03-08T23:08:30.378 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:30.378848 I | op-osd: OSD will have its main bluestore block on "smithi077-lv-1-data-0hb6kc" 2022-03-08T23:08:32.559 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:32.559986 I | op-osd: OSD orchestration status for node smithi106-lv-1-data-0j5llj is "orchestrating" 2022-03-08T23:08:32.650 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:32.651171 I | op-osd: OSD orchestration status for PVC smithi106-lv-1-data-0j5llj is "orchestrating" 2022-03-08T23:08:33.731 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:33.731969 I | op-osd: OSD orchestration status for PVC smithi106-lv-1-data-0j5llj is "completed" 2022-03-08T23:08:33.732 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:33.732001 I | op-osd: creating OSD 11 on PVC "smithi106-lv-1-data-0j5llj" 2022-03-08T23:08:33.732 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:33.732014 I | op-osd: OSD will have its main bluestore block on "smithi106-lv-1-data-0j5llj" 2022-03-08T23:08:34.364 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:34.365135 I | op-osd: finished running OSDs in namespace "rook-ceph" 2022-03-08T23:08:34.365 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:08:34.365162 I | ceph-cluster-controller: done reconciling ceph cluster in namespace "rook-ceph" 2022-03-08T23:08:39.514 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph osd stat -f json 2022-03-08T23:08:40.329 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-03-08T23:08:40.359 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":32,"num_osds":12,"num_up_osds":11,"osd_up_since":1646780899,"num_in_osds":12,"osd_in_since":1646780858,"num_remapped_pgs":0} 2022-03-08T23:08:40.360 INFO:tasks.rook: have 11/12 OSDs 2022-03-08T23:08:50.361 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph osd stat -f json 2022-03-08T23:08:50.876 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-03-08T23:08:50.904 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":32,"num_osds":12,"num_up_osds":11,"osd_up_since":1646780899,"num_in_osds":12,"osd_in_since":1646780858,"num_remapped_pgs":0} 2022-03-08T23:08:50.905 INFO:tasks.rook: have 11/12 OSDs 2022-03-08T23:09:00.906 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph osd stat -f json 2022-03-08T23:09:01.423 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-03-08T23:09:01.450 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":36,"num_osds":12,"num_up_osds":12,"osd_up_since":1646780934,"num_in_osds":12,"osd_in_since":1646780858,"num_remapped_pgs":0} 2022-03-08T23:09:01.451 INFO:tasks.rook:Distributing ceph config and client.admin keyring 2022-03-08T23:09:01.452 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- cat /etc/ceph/ceph.conf 2022-03-08T23:09:01.633 INFO:teuthology.orchestra.run.smithi077.stdout:[global] 2022-03-08T23:09:01.634 INFO:teuthology.orchestra.run.smithi077.stdout:mon_host = 10.110.141.123:6789,10.98.41.230:6789,10.109.47.142:6789 2022-03-08T23:09:01.634 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-03-08T23:09:01.635 INFO:teuthology.orchestra.run.smithi077.stdout:[client.admin] 2022-03-08T23:09:01.635 INFO:teuthology.orchestra.run.smithi077.stdout:keyring = /etc/ceph/keyring 2022-03-08T23:09:01.639 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- cat /etc/ceph/keyring 2022-03-08T23:09:01.819 INFO:teuthology.orchestra.run.smithi077.stdout:[client.admin] 2022-03-08T23:09:01.820 INFO:teuthology.orchestra.run.smithi077.stdout:key = AQC04CdiQV8aEBAAW6Y932PSf7422C1tR4tHgQ== 2022-03-08T23:09:01.824 DEBUG:teuthology.orchestra.run.smithi077:> sudo mkdir -p /etc/ceph 2022-03-08T23:09:01.839 DEBUG:teuthology.orchestra.run.smithi106:> sudo mkdir -p /etc/ceph 2022-03-08T23:09:01.854 DEBUG:teuthology.orchestra.run.smithi135:> sudo mkdir -p /etc/ceph 2022-03-08T23:09:01.869 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2022-03-08T23:09:01.869 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/etc/ceph/ceph.conf 2022-03-08T23:09:01.894 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2022-03-08T23:09:01.894 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/etc/ceph/keyring 2022-03-08T23:09:01.906 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2022-03-08T23:09:01.907 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd of=/etc/ceph/ceph.conf 2022-03-08T23:09:01.924 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2022-03-08T23:09:01.924 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd of=/etc/ceph/keyring 2022-03-08T23:09:01.937 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2022-03-08T23:09:01.937 DEBUG:teuthology.orchestra.run.smithi135:> sudo dd of=/etc/ceph/ceph.conf 2022-03-08T23:09:01.953 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2022-03-08T23:09:01.953 DEBUG:teuthology.orchestra.run.smithi135:> sudo dd of=/etc/ceph/keyring 2022-03-08T23:09:01.966 INFO:tasks.rook:Setting up client nodes... 2022-03-08T23:09:01.967 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph auth get-or-create client.a mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2022-03-08T23:09:02.730 INFO:teuthology.orchestra.run.smithi077.stdout:[client.a] 2022-03-08T23:09:02.731 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQAO4idi2gTLIhAAMWJTeO1mPArr1++rqRXT8w== 2022-03-08T23:09:02.757 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2022-03-08T23:09:02.757 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/etc/ceph/ceph.client.a.keyring 2022-03-08T23:09:02.757 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod 0644 /etc/ceph/ceph.client.a.keyring 2022-03-08T23:09:02.778 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph auth get-or-create client.b mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2022-03-08T23:09:03.440 INFO:teuthology.orchestra.run.smithi077.stdout:[client.b] 2022-03-08T23:09:03.440 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQAP4idigN0LFBAAZAmOoxJvxEypzKkF3O/kQQ== 2022-03-08T23:09:03.466 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2022-03-08T23:09:03.466 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd of=/etc/ceph/ceph.client.b.keyring 2022-03-08T23:09:03.467 DEBUG:teuthology.orchestra.run.smithi106:> sudo chmod 0644 /etc/ceph/ceph.client.b.keyring 2022-03-08T23:09:03.492 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph auth get-or-create client.c mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2022-03-08T23:09:04.325 INFO:teuthology.orchestra.run.smithi077.stdout:[client.c] 2022-03-08T23:09:04.326 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQAQ4idiQZ2+BxAA8X4bJKuy72Oe1Yg1efSGfQ== 2022-03-08T23:09:04.353 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2022-03-08T23:09:04.354 DEBUG:teuthology.orchestra.run.smithi135:> sudo dd of=/etc/ceph/ceph.client.c.keyring 2022-03-08T23:09:04.354 DEBUG:teuthology.orchestra.run.smithi135:> sudo chmod 0644 /etc/ceph/ceph.client.c.keyring 2022-03-08T23:09:04.378 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph osd dump --format=json 2022-03-08T23:09:04.939 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-03-08T23:09:04.940 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":36,"fsid":"965c96f5-6497-4297-a748-9f1a68fefb7b","created":"2022-03-08T23:03:36.157091+0000","modified":"2022-03-08T23:08:55.840477+0000","last_up_change":"2022-03-08T23:08:54.662879+0000","last_in_change":"2022-03-08T23:07:38.978898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":16,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-03-08T23:08:08.183737+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"on","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"a69664c2-477b-4fea-b59c-3aac839d8627","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.250.96.11:6800","nonce":3772645745},{"type":"v1","addr":"10.250.96.11:6801","nonce":3772645745}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.250.96.11:6802","nonce":3772645745},{"type":"v1","addr":"10.250.96.11:6803","nonce":3772645745}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.250.96.11:6806","nonce":3772645745},{"type":"v1","addr":"10.250.96.11:6807","nonce":3772645745}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.250.96.11:6804","nonce":3772645745},{"type":"v1","addr":"10.250.96.11:6805","nonce":3772645745}]},"public_addr":"10.250.96.11:6801/3772645745","cluster_addr":"10.250.96.11:6803/3772645745","heartbeat_back_addr":"10.250.96.11:6807/3772645745","heartbeat_front_addr":"10.250.96.11:6805/3772645745","state":["exists","up"]},{"osd":1,"uuid":"5bad79a1-2ec5-4c5d-af05-f73f3c069fc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.250.97.18:6800","nonce":2876779621},{"type":"v1","addr":"10.250.97.18:6801","nonce":2876779621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.250.97.18:6802","nonce":2876779621},{"type":"v1","addr":"10.250.97.18:6803","nonce":2876779621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.250.97.18:6806","nonce":2876779621},{"type":"v1","addr":"10.250.97.18:6807","nonce":2876779621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.250.97.18:6804","nonce":2876779621},{"type":"v1","addr":"10.250.97.18:6805","nonce":2876779621}]},"public_addr":"10.250.97.18:6801/2876779621","cluster_addr":"10.250.97.18:6803/2876779621","heartbeat_back_addr":"10.250.97.18:6807/2876779621","heartbeat_front_addr":"10.250.97.18:6805/2876779621","state":["exists","up"]},{"osd":2,"uuid":"c711d86f-e842-49d9-956c-82d02057107f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.250.97.16:6800","nonce":318206100},{"type":"v1","addr":"10.250.97.16:6801","nonce":318206100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.250.97.16:6802","nonce":318206100},{"type":"v1","addr":"10.250.97.16:6803","nonce":318206100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.250.97.16:6806","nonce":318206100},{"type":"v1","addr":"10.250.97.16:6807","nonce":318206100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.250.97.16:6804","nonce":318206100},{"type":"v1","addr":"10.250.97.16:6805","nonce":318206100}]},"public_addr":"10.250.97.16:6801/318206100","cluster_addr":"10.250.97.16:6803/318206100","heartbeat_back_addr":"10.250.97.16:6807/318206100","heartbeat_front_addr":"10.250.97.16:6805/318206100","state":["exists","up"]},{"osd":3,"uuid":"1a530686-d4c3-46b1-8f71-4411f6fca784","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.250.98.13:6800","nonce":2849695509},{"type":"v1","addr":"10.250.98.13:6801","nonce":2849695509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.250.98.13:6802","nonce":2849695509},{"type":"v1","addr":"10.250.98.13:6803","nonce":2849695509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.250.98.13:6806","nonce":2849695509},{"type":"v1","addr":"10.250.98.13:6807","nonce":2849695509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.250.98.13:6804","nonce":2849695509},{"type":"v1","addr":"10.250.98.13:6805","nonce":2849695509}]},"public_addr":"10.250.98.13:6801/2849695509","cluster_addr":"10.250.98.13:6803/2849695509","heartbeat_back_addr":"10.250.98.13:6807/2849695509","heartbeat_front_addr":"10.250.98.13:6805/2849695509","state":["exists","up"]},{"osd":4,"uuid":"94461549-ebb5-409b-be9e-43311347943b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.250.98.14:6800","nonce":2984935157},{"type":"v1","addr":"10.250.98.14:6801","nonce":2984935157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.250.98.14:6802","nonce":2984935157},{"type":"v1","addr":"10.250.98.14:6803","nonce":2984935157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.250.98.14:6806","nonce":2984935157},{"type":"v1","addr":"10.250.98.14:6807","nonce":2984935157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.250.98.14:6804","nonce":2984935157},{"type":"v1","addr":"10.250.98.14:6805","nonce":2984935157}]},"public_addr":"10.250.98.14:6801/2984935157","cluster_addr":"10.250.98.14:6803/2984935157","heartbeat_back_addr":"10.250.98.14:6807/2984935157","heartbeat_front_addr":"10.250.98.14:6805/2984935157","state":["exists","up"]},{"osd":5,"uuid":"8d771635-6b9e-498a-9a9d-3b7279afd009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":28,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.250.96.12:6800","nonce":529128875},{"type":"v1","addr":"10.250.96.12:6801","nonce":529128875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.250.96.12:6802","nonce":529128875},{"type":"v1","addr":"10.250.96.12:6803","nonce":529128875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.250.96.12:6806","nonce":529128875},{"type":"v1","addr":"10.250.96.12:6807","nonce":529128875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.250.96.12:6804","nonce":529128875},{"type":"v1","addr":"10.250.96.12:6805","nonce":529128875}]},"public_addr":"10.250.96.12:6801/529128875","cluster_addr":"10.250.96.12:6803/529128875","heartbeat_back_addr":"10.250.96.12:6807/529128875","heartbeat_front_addr":"10.250.96.12:6805/529128875","state":["exists","up"]},{"osd":6,"uuid":"45749ee6-c26c-4e4e-aed9-11f7f9931ce2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.250.98.15:6800","nonce":100968526},{"type":"v1","addr":"10.250.98.15:6801","nonce":100968526}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.250.98.15:6802","nonce":100968526},{"type":"v1","addr":"10.250.98.15:6803","nonce":100968526}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.250.98.15:6806","nonce":100968526},{"type":"v1","addr":"10.250.98.15:6807","nonce":100968526}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.250.98.15:6804","nonce":100968526},{"type":"v1","addr":"10.250.98.15:6805","nonce":100968526}]},"public_addr":"10.250.98.15:6801/100968526","cluster_addr":"10.250.98.15:6803/100968526","heartbeat_back_addr":"10.250.98.15:6807/100968526","heartbeat_front_addr":"10.250.98.15:6805/100968526","state":["exists","up"]},{"osd":7,"uuid":"5713d388-dacc-4f36-bd81-a49555b177aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.250.98.16:6800","nonce":832955710},{"type":"v1","addr":"10.250.98.16:6801","nonce":832955710}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.250.98.16:6802","nonce":832955710},{"type":"v1","addr":"10.250.98.16:6803","nonce":832955710}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.250.98.16:6806","nonce":832955710},{"type":"v1","addr":"10.250.98.16:6807","nonce":832955710}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.250.98.16:6804","nonce":832955710},{"type":"v1","addr":"10.250.98.16:6805","nonce":832955710}]},"public_addr":"10.250.98.16:6801/832955710","cluster_addr":"10.250.98.16:6803/832955710","heartbeat_back_addr":"10.250.98.16:6807/832955710","heartbeat_front_addr":"10.250.98.16:6805/832955710","state":["exists","up"]},{"osd":8,"uuid":"50561567-8845-434d-80f9-9f748fa099dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.250.97.19:6800","nonce":1740384438},{"type":"v1","addr":"10.250.97.19:6801","nonce":1740384438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.250.97.19:6802","nonce":1740384438},{"type":"v1","addr":"10.250.97.19:6803","nonce":1740384438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.250.97.19:6806","nonce":1740384438},{"type":"v1","addr":"10.250.97.19:6807","nonce":1740384438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.250.97.19:6804","nonce":1740384438},{"type":"v1","addr":"10.250.97.19:6805","nonce":1740384438}]},"public_addr":"10.250.97.19:6801/1740384438","cluster_addr":"10.250.97.19:6803/1740384438","heartbeat_back_addr":"10.250.97.19:6807/1740384438","heartbeat_front_addr":"10.250.97.19:6805/1740384438","state":["exists","up"]},{"osd":9,"uuid":"5e3101bc-1423-44fc-a972-d83163165889","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.250.96.13:6800","nonce":3518030366},{"type":"v1","addr":"10.250.96.13:6801","nonce":3518030366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.250.96.13:6802","nonce":3518030366},{"type":"v1","addr":"10.250.96.13:6803","nonce":3518030366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.250.96.13:6806","nonce":3518030366},{"type":"v1","addr":"10.250.96.13:6807","nonce":3518030366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.250.96.13:6804","nonce":3518030366},{"type":"v1","addr":"10.250.96.13:6805","nonce":3518030366}]},"public_addr":"10.250.96.13:6801/3518030366","cluster_addr":"10.250.96.13:6803/3518030366","heartbeat_back_addr":"10.250.96.13:6807/3518030366","heartbeat_front_addr":"10.250.96.13:6805/3518030366","state":["exists","up"]},{"osd":10,"uuid":"eed97238-0880-45f0-880c-46d7b5af1bfa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.250.96.14:6800","nonce":1505204262},{"type":"v1","addr":"10.250.96.14:6801","nonce":1505204262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.250.96.14:6802","nonce":1505204262},{"type":"v1","addr":"10.250.96.14:6803","nonce":1505204262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.250.96.14:6806","nonce":1505204262},{"type":"v1","addr":"10.250.96.14:6807","nonce":1505204262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.250.96.14:6804","nonce":1505204262},{"type":"v1","addr":"10.250.96.14:6805","nonce":1505204262}]},"public_addr":"10.250.96.14:6801/1505204262","cluster_addr":"10.250.96.14:6803/1505204262","heartbeat_back_addr":"10.250.96.14:6807/1505204262","heartbeat_front_addr":"10.250.96.14:6805/1505204262","state":["exists","up"]},{"osd":11,"uuid":"1efc93ab-b29d-4c75-88bb-5c1dd4900519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.250.97.21:6800","nonce":162842836},{"type":"v1","addr":"10.250.97.21:6801","nonce":162842836}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.250.97.21:6802","nonce":162842836},{"type":"v1","addr":"10.250.97.21:6803","nonce":162842836}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.250.97.21:6806","nonce":162842836},{"type":"v1","addr":"10.250.97.21:6807","nonce":162842836}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.250.97.21:6804","nonce":162842836},{"type":"v1","addr":"10.250.97.21:6805","nonce":162842836}]},"public_addr":"10.250.97.21:6801/162842836","cluster_addr":"10.250.97.21:6803/162842836","heartbeat_back_addr":"10.250.97.21:6807/162842836","heartbeat_front_addr":"10.250.97.21:6805/162842836","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T23:07:55.470249+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T23:08:01.238763+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T23:08:01.223564+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T23:08:00.077239+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T23:08:00.223752+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T23:08:13.107171+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T23:08:02.995546+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T23:08:05.415113+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T23:08:07.294830+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T23:08:16.679213+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T23:08:18.083421+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T23:08:52.128206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.250.97.10:0/2737284081":"2022-03-09T23:06:32.835329+0000","10.250.97.10:6801/3324143941":"2022-03-09T23:06:32.835329+0000","10.250.97.10:0/2736804224":"2022-03-09T23:06:32.835329+0000","10.250.97.10:6800/3324143941":"2022-03-09T23:06:32.835329+0000","10.250.97.10:0/1820152328":"2022-03-09T23:06:32.835329+0000"},"erasure_code_profiles":{"default":{"k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-03-08T23:09:04.958 INFO:tasks.rook.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2022-03-08T23:08:08.183737+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'on', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2022-03-08T23:09:04.959 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph osd pool get .mgr pg_num 2022-03-08T23:09:05.463 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 1 2022-03-08T23:09:05.483 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2022-03-08T23:09:05.484 INFO:tasks.rook.ceph_manager.ceph:waiting for mgr available 2022-03-08T23:09:05.485 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph mgr dump --format=json 2022-03-08T23:09:06.102 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-03-08T23:09:06.121 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":14,"active_gid":14287,"active_name":"a","active_addrs":{"addrvec":[{"type":"v2","addr":"10.250.97.10:6800","nonce":656737707},{"type":"v1","addr":"10.250.97.10:6801","nonce":656737707}]},"active_addr":"10.250.97.10:6801/656737707","active_change":"2022-03-08T23:06:32.865039+0000","active_mgr_features":4540138303579357183,"available":true,"standbys":[],"modules":["iostat","nfs","prometheus","restful","rook"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:6.7.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"0.0.0.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"0.0.0.0","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"prometheus":"http://10.250.97.10:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":2,"active_clients":[{"addrvec":[{"type":"v2","addr":"10.250.97.10:0","nonce":2342036079}]},{"addrvec":[{"type":"v2","addr":"10.250.97.10:0","nonce":3026924508}]},{"addrvec":[{"type":"v2","addr":"10.250.97.10:0","nonce":607322542}]},{"addrvec":[{"type":"v2","addr":"10.250.97.10:0","nonce":423288587}]}]}} 2022-03-08T23:09:06.124 INFO:tasks.rook.ceph_manager.ceph:mgr available! 2022-03-08T23:09:06.124 INFO:tasks.rook.ceph_manager.ceph:waiting for all up 2022-03-08T23:09:06.125 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph osd dump --format=json 2022-03-08T23:09:06.647 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-03-08T23:09:06.648 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":36,"fsid":"965c96f5-6497-4297-a748-9f1a68fefb7b","created":"2022-03-08T23:03:36.157091+0000","modified":"2022-03-08T23:08:55.840477+0000","last_up_change":"2022-03-08T23:08:54.662879+0000","last_in_change":"2022-03-08T23:07:38.978898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":16,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-03-08T23:08:08.183737+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"on","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"a69664c2-477b-4fea-b59c-3aac839d8627","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.250.96.11:6800","nonce":3772645745},{"type":"v1","addr":"10.250.96.11:6801","nonce":3772645745}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.250.96.11:6802","nonce":3772645745},{"type":"v1","addr":"10.250.96.11:6803","nonce":3772645745}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.250.96.11:6806","nonce":3772645745},{"type":"v1","addr":"10.250.96.11:6807","nonce":3772645745}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.250.96.11:6804","nonce":3772645745},{"type":"v1","addr":"10.250.96.11:6805","nonce":3772645745}]},"public_addr":"10.250.96.11:6801/3772645745","cluster_addr":"10.250.96.11:6803/3772645745","heartbeat_back_addr":"10.250.96.11:6807/3772645745","heartbeat_front_addr":"10.250.96.11:6805/3772645745","state":["exists","up"]},{"osd":1,"uuid":"5bad79a1-2ec5-4c5d-af05-f73f3c069fc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.250.97.18:6800","nonce":2876779621},{"type":"v1","addr":"10.250.97.18:6801","nonce":2876779621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.250.97.18:6802","nonce":2876779621},{"type":"v1","addr":"10.250.97.18:6803","nonce":2876779621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.250.97.18:6806","nonce":2876779621},{"type":"v1","addr":"10.250.97.18:6807","nonce":2876779621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.250.97.18:6804","nonce":2876779621},{"type":"v1","addr":"10.250.97.18:6805","nonce":2876779621}]},"public_addr":"10.250.97.18:6801/2876779621","cluster_addr":"10.250.97.18:6803/2876779621","heartbeat_back_addr":"10.250.97.18:6807/2876779621","heartbeat_front_addr":"10.250.97.18:6805/2876779621","state":["exists","up"]},{"osd":2,"uuid":"c711d86f-e842-49d9-956c-82d02057107f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.250.97.16:6800","nonce":318206100},{"type":"v1","addr":"10.250.97.16:6801","nonce":318206100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.250.97.16:6802","nonce":318206100},{"type":"v1","addr":"10.250.97.16:6803","nonce":318206100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.250.97.16:6806","nonce":318206100},{"type":"v1","addr":"10.250.97.16:6807","nonce":318206100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.250.97.16:6804","nonce":318206100},{"type":"v1","addr":"10.250.97.16:6805","nonce":318206100}]},"public_addr":"10.250.97.16:6801/318206100","cluster_addr":"10.250.97.16:6803/318206100","heartbeat_back_addr":"10.250.97.16:6807/318206100","heartbeat_front_addr":"10.250.97.16:6805/318206100","state":["exists","up"]},{"osd":3,"uuid":"1a530686-d4c3-46b1-8f71-4411f6fca784","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.250.98.13:6800","nonce":2849695509},{"type":"v1","addr":"10.250.98.13:6801","nonce":2849695509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.250.98.13:6802","nonce":2849695509},{"type":"v1","addr":"10.250.98.13:6803","nonce":2849695509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.250.98.13:6806","nonce":2849695509},{"type":"v1","addr":"10.250.98.13:6807","nonce":2849695509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.250.98.13:6804","nonce":2849695509},{"type":"v1","addr":"10.250.98.13:6805","nonce":2849695509}]},"public_addr":"10.250.98.13:6801/2849695509","cluster_addr":"10.250.98.13:6803/2849695509","heartbeat_back_addr":"10.250.98.13:6807/2849695509","heartbeat_front_addr":"10.250.98.13:6805/2849695509","state":["exists","up"]},{"osd":4,"uuid":"94461549-ebb5-409b-be9e-43311347943b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.250.98.14:6800","nonce":2984935157},{"type":"v1","addr":"10.250.98.14:6801","nonce":2984935157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.250.98.14:6802","nonce":2984935157},{"type":"v1","addr":"10.250.98.14:6803","nonce":2984935157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.250.98.14:6806","nonce":2984935157},{"type":"v1","addr":"10.250.98.14:6807","nonce":2984935157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.250.98.14:6804","nonce":2984935157},{"type":"v1","addr":"10.250.98.14:6805","nonce":2984935157}]},"public_addr":"10.250.98.14:6801/2984935157","cluster_addr":"10.250.98.14:6803/2984935157","heartbeat_back_addr":"10.250.98.14:6807/2984935157","heartbeat_front_addr":"10.250.98.14:6805/2984935157","state":["exists","up"]},{"osd":5,"uuid":"8d771635-6b9e-498a-9a9d-3b7279afd009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":28,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.250.96.12:6800","nonce":529128875},{"type":"v1","addr":"10.250.96.12:6801","nonce":529128875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.250.96.12:6802","nonce":529128875},{"type":"v1","addr":"10.250.96.12:6803","nonce":529128875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.250.96.12:6806","nonce":529128875},{"type":"v1","addr":"10.250.96.12:6807","nonce":529128875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.250.96.12:6804","nonce":529128875},{"type":"v1","addr":"10.250.96.12:6805","nonce":529128875}]},"public_addr":"10.250.96.12:6801/529128875","cluster_addr":"10.250.96.12:6803/529128875","heartbeat_back_addr":"10.250.96.12:6807/529128875","heartbeat_front_addr":"10.250.96.12:6805/529128875","state":["exists","up"]},{"osd":6,"uuid":"45749ee6-c26c-4e4e-aed9-11f7f9931ce2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.250.98.15:6800","nonce":100968526},{"type":"v1","addr":"10.250.98.15:6801","nonce":100968526}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.250.98.15:6802","nonce":100968526},{"type":"v1","addr":"10.250.98.15:6803","nonce":100968526}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.250.98.15:6806","nonce":100968526},{"type":"v1","addr":"10.250.98.15:6807","nonce":100968526}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.250.98.15:6804","nonce":100968526},{"type":"v1","addr":"10.250.98.15:6805","nonce":100968526}]},"public_addr":"10.250.98.15:6801/100968526","cluster_addr":"10.250.98.15:6803/100968526","heartbeat_back_addr":"10.250.98.15:6807/100968526","heartbeat_front_addr":"10.250.98.15:6805/100968526","state":["exists","up"]},{"osd":7,"uuid":"5713d388-dacc-4f36-bd81-a49555b177aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.250.98.16:6800","nonce":832955710},{"type":"v1","addr":"10.250.98.16:6801","nonce":832955710}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.250.98.16:6802","nonce":832955710},{"type":"v1","addr":"10.250.98.16:6803","nonce":832955710}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.250.98.16:6806","nonce":832955710},{"type":"v1","addr":"10.250.98.16:6807","nonce":832955710}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.250.98.16:6804","nonce":832955710},{"type":"v1","addr":"10.250.98.16:6805","nonce":832955710}]},"public_addr":"10.250.98.16:6801/832955710","cluster_addr":"10.250.98.16:6803/832955710","heartbeat_back_addr":"10.250.98.16:6807/832955710","heartbeat_front_addr":"10.250.98.16:6805/832955710","state":["exists","up"]},{"osd":8,"uuid":"50561567-8845-434d-80f9-9f748fa099dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.250.97.19:6800","nonce":1740384438},{"type":"v1","addr":"10.250.97.19:6801","nonce":1740384438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.250.97.19:6802","nonce":1740384438},{"type":"v1","addr":"10.250.97.19:6803","nonce":1740384438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.250.97.19:6806","nonce":1740384438},{"type":"v1","addr":"10.250.97.19:6807","nonce":1740384438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.250.97.19:6804","nonce":1740384438},{"type":"v1","addr":"10.250.97.19:6805","nonce":1740384438}]},"public_addr":"10.250.97.19:6801/1740384438","cluster_addr":"10.250.97.19:6803/1740384438","heartbeat_back_addr":"10.250.97.19:6807/1740384438","heartbeat_front_addr":"10.250.97.19:6805/1740384438","state":["exists","up"]},{"osd":9,"uuid":"5e3101bc-1423-44fc-a972-d83163165889","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.250.96.13:6800","nonce":3518030366},{"type":"v1","addr":"10.250.96.13:6801","nonce":3518030366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.250.96.13:6802","nonce":3518030366},{"type":"v1","addr":"10.250.96.13:6803","nonce":3518030366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.250.96.13:6806","nonce":3518030366},{"type":"v1","addr":"10.250.96.13:6807","nonce":3518030366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.250.96.13:6804","nonce":3518030366},{"type":"v1","addr":"10.250.96.13:6805","nonce":3518030366}]},"public_addr":"10.250.96.13:6801/3518030366","cluster_addr":"10.250.96.13:6803/3518030366","heartbeat_back_addr":"10.250.96.13:6807/3518030366","heartbeat_front_addr":"10.250.96.13:6805/3518030366","state":["exists","up"]},{"osd":10,"uuid":"eed97238-0880-45f0-880c-46d7b5af1bfa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.250.96.14:6800","nonce":1505204262},{"type":"v1","addr":"10.250.96.14:6801","nonce":1505204262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.250.96.14:6802","nonce":1505204262},{"type":"v1","addr":"10.250.96.14:6803","nonce":1505204262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.250.96.14:6806","nonce":1505204262},{"type":"v1","addr":"10.250.96.14:6807","nonce":1505204262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.250.96.14:6804","nonce":1505204262},{"type":"v1","addr":"10.250.96.14:6805","nonce":1505204262}]},"public_addr":"10.250.96.14:6801/1505204262","cluster_addr":"10.250.96.14:6803/1505204262","heartbeat_back_addr":"10.250.96.14:6807/1505204262","heartbeat_front_addr":"10.250.96.14:6805/1505204262","state":["exists","up"]},{"osd":11,"uuid":"1efc93ab-b29d-4c75-88bb-5c1dd4900519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.250.97.21:6800","nonce":162842836},{"type":"v1","addr":"10.250.97.21:6801","nonce":162842836}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.250.97.21:6802","nonce":162842836},{"type":"v1","addr":"10.250.97.21:6803","nonce":162842836}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.250.97.21:6806","nonce":162842836},{"type":"v1","addr":"10.250.97.21:6807","nonce":162842836}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.250.97.21:6804","nonce":162842836},{"type":"v1","addr":"10.250.97.21:6805","nonce":162842836}]},"public_addr":"10.250.97.21:6801/162842836","cluster_addr":"10.250.97.21:6803/162842836","heartbeat_back_addr":"10.250.97.21:6807/162842836","heartbeat_front_addr":"10.250.97.21:6805/162842836","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T23:07:55.470249+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T23:08:01.238763+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T23:08:01.223564+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T23:08:00.077239+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T23:08:00.223752+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T23:08:13.107171+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T23:08:02.995546+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T23:08:05.415113+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T23:08:07.294830+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T23:08:16.679213+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T23:08:18.083421+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T23:08:52.128206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.250.97.10:0/2737284081":"2022-03-09T23:06:32.835329+0000","10.250.97.10:6801/3324143941":"2022-03-09T23:06:32.835329+0000","10.250.97.10:0/2736804224":"2022-03-09T23:06:32.835329+0000","10.250.97.10:6800/3324143941":"2022-03-09T23:06:32.835329+0000","10.250.97.10:0/1820152328":"2022-03-09T23:06:32.835329+0000"},"erasure_code_profiles":{"default":{"k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-03-08T23:09:06.667 INFO:tasks.rook.ceph_manager.ceph:all up! 2022-03-08T23:09:06.667 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph osd dump --format=json 2022-03-08T23:09:07.203 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-03-08T23:09:07.203 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":36,"fsid":"965c96f5-6497-4297-a748-9f1a68fefb7b","created":"2022-03-08T23:03:36.157091+0000","modified":"2022-03-08T23:08:55.840477+0000","last_up_change":"2022-03-08T23:08:54.662879+0000","last_in_change":"2022-03-08T23:07:38.978898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":16,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-03-08T23:08:08.183737+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"on","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"a69664c2-477b-4fea-b59c-3aac839d8627","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.250.96.11:6800","nonce":3772645745},{"type":"v1","addr":"10.250.96.11:6801","nonce":3772645745}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.250.96.11:6802","nonce":3772645745},{"type":"v1","addr":"10.250.96.11:6803","nonce":3772645745}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.250.96.11:6806","nonce":3772645745},{"type":"v1","addr":"10.250.96.11:6807","nonce":3772645745}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.250.96.11:6804","nonce":3772645745},{"type":"v1","addr":"10.250.96.11:6805","nonce":3772645745}]},"public_addr":"10.250.96.11:6801/3772645745","cluster_addr":"10.250.96.11:6803/3772645745","heartbeat_back_addr":"10.250.96.11:6807/3772645745","heartbeat_front_addr":"10.250.96.11:6805/3772645745","state":["exists","up"]},{"osd":1,"uuid":"5bad79a1-2ec5-4c5d-af05-f73f3c069fc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.250.97.18:6800","nonce":2876779621},{"type":"v1","addr":"10.250.97.18:6801","nonce":2876779621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.250.97.18:6802","nonce":2876779621},{"type":"v1","addr":"10.250.97.18:6803","nonce":2876779621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.250.97.18:6806","nonce":2876779621},{"type":"v1","addr":"10.250.97.18:6807","nonce":2876779621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.250.97.18:6804","nonce":2876779621},{"type":"v1","addr":"10.250.97.18:6805","nonce":2876779621}]},"public_addr":"10.250.97.18:6801/2876779621","cluster_addr":"10.250.97.18:6803/2876779621","heartbeat_back_addr":"10.250.97.18:6807/2876779621","heartbeat_front_addr":"10.250.97.18:6805/2876779621","state":["exists","up"]},{"osd":2,"uuid":"c711d86f-e842-49d9-956c-82d02057107f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.250.97.16:6800","nonce":318206100},{"type":"v1","addr":"10.250.97.16:6801","nonce":318206100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.250.97.16:6802","nonce":318206100},{"type":"v1","addr":"10.250.97.16:6803","nonce":318206100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.250.97.16:6806","nonce":318206100},{"type":"v1","addr":"10.250.97.16:6807","nonce":318206100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.250.97.16:6804","nonce":318206100},{"type":"v1","addr":"10.250.97.16:6805","nonce":318206100}]},"public_addr":"10.250.97.16:6801/318206100","cluster_addr":"10.250.97.16:6803/318206100","heartbeat_back_addr":"10.250.97.16:6807/318206100","heartbeat_front_addr":"10.250.97.16:6805/318206100","state":["exists","up"]},{"osd":3,"uuid":"1a530686-d4c3-46b1-8f71-4411f6fca784","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.250.98.13:6800","nonce":2849695509},{"type":"v1","addr":"10.250.98.13:6801","nonce":2849695509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.250.98.13:6802","nonce":2849695509},{"type":"v1","addr":"10.250.98.13:6803","nonce":2849695509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.250.98.13:6806","nonce":2849695509},{"type":"v1","addr":"10.250.98.13:6807","nonce":2849695509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.250.98.13:6804","nonce":2849695509},{"type":"v1","addr":"10.250.98.13:6805","nonce":2849695509}]},"public_addr":"10.250.98.13:6801/2849695509","cluster_addr":"10.250.98.13:6803/2849695509","heartbeat_back_addr":"10.250.98.13:6807/2849695509","heartbeat_front_addr":"10.250.98.13:6805/2849695509","state":["exists","up"]},{"osd":4,"uuid":"94461549-ebb5-409b-be9e-43311347943b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.250.98.14:6800","nonce":2984935157},{"type":"v1","addr":"10.250.98.14:6801","nonce":2984935157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.250.98.14:6802","nonce":2984935157},{"type":"v1","addr":"10.250.98.14:6803","nonce":2984935157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.250.98.14:6806","nonce":2984935157},{"type":"v1","addr":"10.250.98.14:6807","nonce":2984935157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.250.98.14:6804","nonce":2984935157},{"type":"v1","addr":"10.250.98.14:6805","nonce":2984935157}]},"public_addr":"10.250.98.14:6801/2984935157","cluster_addr":"10.250.98.14:6803/2984935157","heartbeat_back_addr":"10.250.98.14:6807/2984935157","heartbeat_front_addr":"10.250.98.14:6805/2984935157","state":["exists","up"]},{"osd":5,"uuid":"8d771635-6b9e-498a-9a9d-3b7279afd009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":28,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.250.96.12:6800","nonce":529128875},{"type":"v1","addr":"10.250.96.12:6801","nonce":529128875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.250.96.12:6802","nonce":529128875},{"type":"v1","addr":"10.250.96.12:6803","nonce":529128875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.250.96.12:6806","nonce":529128875},{"type":"v1","addr":"10.250.96.12:6807","nonce":529128875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.250.96.12:6804","nonce":529128875},{"type":"v1","addr":"10.250.96.12:6805","nonce":529128875}]},"public_addr":"10.250.96.12:6801/529128875","cluster_addr":"10.250.96.12:6803/529128875","heartbeat_back_addr":"10.250.96.12:6807/529128875","heartbeat_front_addr":"10.250.96.12:6805/529128875","state":["exists","up"]},{"osd":6,"uuid":"45749ee6-c26c-4e4e-aed9-11f7f9931ce2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.250.98.15:6800","nonce":100968526},{"type":"v1","addr":"10.250.98.15:6801","nonce":100968526}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.250.98.15:6802","nonce":100968526},{"type":"v1","addr":"10.250.98.15:6803","nonce":100968526}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.250.98.15:6806","nonce":100968526},{"type":"v1","addr":"10.250.98.15:6807","nonce":100968526}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.250.98.15:6804","nonce":100968526},{"type":"v1","addr":"10.250.98.15:6805","nonce":100968526}]},"public_addr":"10.250.98.15:6801/100968526","cluster_addr":"10.250.98.15:6803/100968526","heartbeat_back_addr":"10.250.98.15:6807/100968526","heartbeat_front_addr":"10.250.98.15:6805/100968526","state":["exists","up"]},{"osd":7,"uuid":"5713d388-dacc-4f36-bd81-a49555b177aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.250.98.16:6800","nonce":832955710},{"type":"v1","addr":"10.250.98.16:6801","nonce":832955710}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.250.98.16:6802","nonce":832955710},{"type":"v1","addr":"10.250.98.16:6803","nonce":832955710}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.250.98.16:6806","nonce":832955710},{"type":"v1","addr":"10.250.98.16:6807","nonce":832955710}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.250.98.16:6804","nonce":832955710},{"type":"v1","addr":"10.250.98.16:6805","nonce":832955710}]},"public_addr":"10.250.98.16:6801/832955710","cluster_addr":"10.250.98.16:6803/832955710","heartbeat_back_addr":"10.250.98.16:6807/832955710","heartbeat_front_addr":"10.250.98.16:6805/832955710","state":["exists","up"]},{"osd":8,"uuid":"50561567-8845-434d-80f9-9f748fa099dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.250.97.19:6800","nonce":1740384438},{"type":"v1","addr":"10.250.97.19:6801","nonce":1740384438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.250.97.19:6802","nonce":1740384438},{"type":"v1","addr":"10.250.97.19:6803","nonce":1740384438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.250.97.19:6806","nonce":1740384438},{"type":"v1","addr":"10.250.97.19:6807","nonce":1740384438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.250.97.19:6804","nonce":1740384438},{"type":"v1","addr":"10.250.97.19:6805","nonce":1740384438}]},"public_addr":"10.250.97.19:6801/1740384438","cluster_addr":"10.250.97.19:6803/1740384438","heartbeat_back_addr":"10.250.97.19:6807/1740384438","heartbeat_front_addr":"10.250.97.19:6805/1740384438","state":["exists","up"]},{"osd":9,"uuid":"5e3101bc-1423-44fc-a972-d83163165889","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.250.96.13:6800","nonce":3518030366},{"type":"v1","addr":"10.250.96.13:6801","nonce":3518030366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.250.96.13:6802","nonce":3518030366},{"type":"v1","addr":"10.250.96.13:6803","nonce":3518030366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.250.96.13:6806","nonce":3518030366},{"type":"v1","addr":"10.250.96.13:6807","nonce":3518030366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.250.96.13:6804","nonce":3518030366},{"type":"v1","addr":"10.250.96.13:6805","nonce":3518030366}]},"public_addr":"10.250.96.13:6801/3518030366","cluster_addr":"10.250.96.13:6803/3518030366","heartbeat_back_addr":"10.250.96.13:6807/3518030366","heartbeat_front_addr":"10.250.96.13:6805/3518030366","state":["exists","up"]},{"osd":10,"uuid":"eed97238-0880-45f0-880c-46d7b5af1bfa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.250.96.14:6800","nonce":1505204262},{"type":"v1","addr":"10.250.96.14:6801","nonce":1505204262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.250.96.14:6802","nonce":1505204262},{"type":"v1","addr":"10.250.96.14:6803","nonce":1505204262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.250.96.14:6806","nonce":1505204262},{"type":"v1","addr":"10.250.96.14:6807","nonce":1505204262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.250.96.14:6804","nonce":1505204262},{"type":"v1","addr":"10.250.96.14:6805","nonce":1505204262}]},"public_addr":"10.250.96.14:6801/1505204262","cluster_addr":"10.250.96.14:6803/1505204262","heartbeat_back_addr":"10.250.96.14:6807/1505204262","heartbeat_front_addr":"10.250.96.14:6805/1505204262","state":["exists","up"]},{"osd":11,"uuid":"1efc93ab-b29d-4c75-88bb-5c1dd4900519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.250.97.21:6800","nonce":162842836},{"type":"v1","addr":"10.250.97.21:6801","nonce":162842836}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.250.97.21:6802","nonce":162842836},{"type":"v1","addr":"10.250.97.21:6803","nonce":162842836}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.250.97.21:6806","nonce":162842836},{"type":"v1","addr":"10.250.97.21:6807","nonce":162842836}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.250.97.21:6804","nonce":162842836},{"type":"v1","addr":"10.250.97.21:6805","nonce":162842836}]},"public_addr":"10.250.97.21:6801/162842836","cluster_addr":"10.250.97.21:6803/162842836","heartbeat_back_addr":"10.250.97.21:6807/162842836","heartbeat_front_addr":"10.250.97.21:6805/162842836","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T23:07:55.470249+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T23:08:01.238763+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T23:08:01.223564+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T23:08:00.077239+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T23:08:00.223752+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T23:08:13.107171+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T23:08:02.995546+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T23:08:05.415113+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T23:08:07.294830+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T23:08:16.679213+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T23:08:18.083421+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T23:08:52.128206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.250.97.10:0/2737284081":"2022-03-09T23:06:32.835329+0000","10.250.97.10:6801/3324143941":"2022-03-09T23:06:32.835329+0000","10.250.97.10:0/2736804224":"2022-03-09T23:06:32.835329+0000","10.250.97.10:6800/3324143941":"2022-03-09T23:06:32.835329+0000","10.250.97.10:0/1820152328":"2022-03-09T23:06:32.835329+0000"},"erasure_code_profiles":{"default":{"k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-03-08T23:09:07.225 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph tell osd.0 flush_pg_stats 2022-03-08T23:09:07.226 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph tell osd.1 flush_pg_stats 2022-03-08T23:09:07.227 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph tell osd.2 flush_pg_stats 2022-03-08T23:09:07.227 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph tell osd.3 flush_pg_stats 2022-03-08T23:09:07.228 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph tell osd.4 flush_pg_stats 2022-03-08T23:09:07.228 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph tell osd.5 flush_pg_stats 2022-03-08T23:09:07.229 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph tell osd.6 flush_pg_stats 2022-03-08T23:09:07.230 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph tell osd.7 flush_pg_stats 2022-03-08T23:09:07.231 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph tell osd.8 flush_pg_stats 2022-03-08T23:09:07.231 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph tell osd.9 flush_pg_stats 2022-03-08T23:09:07.232 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph tell osd.10 flush_pg_stats 2022-03-08T23:09:07.233 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph tell osd.11 flush_pg_stats 2022-03-08T23:09:07.858 INFO:teuthology.orchestra.run.smithi077.stdout:94489280527 2022-03-08T23:09:07.858 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph osd last-stat-seq osd.2 2022-03-08T23:09:08.205 INFO:teuthology.orchestra.run.smithi077.stdout:120259084301 2022-03-08T23:09:08.205 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph osd last-stat-seq osd.5 2022-03-08T23:09:08.209 INFO:teuthology.orchestra.run.smithi077.stdout:94489280525 2022-03-08T23:09:08.209 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph osd last-stat-seq osd.7 2022-03-08T23:09:08.211 INFO:teuthology.orchestra.run.smithi077.stdout:85899345938 2022-03-08T23:09:08.211 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph osd last-stat-seq osd.3 2022-03-08T23:09:08.214 INFO:teuthology.orchestra.run.smithi077.stdout:90194313231 2022-03-08T23:09:08.215 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph osd last-stat-seq osd.6 2022-03-08T23:09:08.217 INFO:teuthology.orchestra.run.smithi077.stdout:73014444050 2022-03-08T23:09:08.217 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph osd last-stat-seq osd.0 2022-03-08T23:09:08.220 INFO:teuthology.orchestra.run.smithi077.stdout:90194313232 2022-03-08T23:09:08.221 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph osd last-stat-seq osd.1 2022-03-08T23:09:08.225 INFO:teuthology.orchestra.run.smithi077.stdout:128849018892 2022-03-08T23:09:08.225 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph osd last-stat-seq osd.9 2022-03-08T23:09:08.235 INFO:teuthology.orchestra.run.smithi077.stdout:98784247823 2022-03-08T23:09:08.241 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph osd last-stat-seq osd.8 2022-03-08T23:09:08.255 INFO:teuthology.orchestra.run.smithi077.stdout:150323855365 2022-03-08T23:09:08.255 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph osd last-stat-seq osd.11 2022-03-08T23:09:08.290 INFO:teuthology.orchestra.run.smithi077.stdout:133143986189 2022-03-08T23:09:08.291 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph osd last-stat-seq osd.10 2022-03-08T23:09:08.402 INFO:teuthology.orchestra.run.smithi077.stdout:81604378640 2022-03-08T23:09:08.402 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph osd last-stat-seq osd.4 2022-03-08T23:09:08.695 INFO:teuthology.orchestra.run.smithi077.stdout:94489280526 2022-03-08T23:09:08.783 INFO:tasks.rook.ceph_manager.ceph:need seq 94489280527 got 94489280526 for osd.2 2022-03-08T23:09:08.993 INFO:teuthology.orchestra.run.smithi077.stdout:94489280524 2022-03-08T23:09:09.050 INFO:tasks.rook.ceph_manager.ceph:need seq 94489280525 got 94489280524 for osd.7 2022-03-08T23:09:09.072 INFO:teuthology.orchestra.run.smithi077.stdout:90194313231 2022-03-08T23:09:09.139 INFO:teuthology.orchestra.run.smithi077.stdout:120259084300 2022-03-08T23:09:09.152 INFO:tasks.rook.ceph_manager.ceph:need seq 90194313232 got 90194313231 for osd.1 2022-03-08T23:09:09.170 INFO:teuthology.orchestra.run.smithi077.stdout:150323855364 2022-03-08T23:09:09.181 INFO:tasks.rook.ceph_manager.ceph:need seq 120259084301 got 120259084300 for osd.5 2022-03-08T23:09:09.213 INFO:tasks.rook.ceph_manager.ceph:need seq 150323855365 got 150323855364 for osd.11 2022-03-08T23:09:09.258 INFO:teuthology.orchestra.run.smithi077.stdout:128849018891 2022-03-08T23:09:09.280 INFO:teuthology.orchestra.run.smithi077.stdout:85899345937 2022-03-08T23:09:09.303 INFO:tasks.rook.ceph_manager.ceph:need seq 128849018892 got 128849018891 for osd.9 2022-03-08T23:09:09.308 INFO:tasks.rook.ceph_manager.ceph:need seq 85899345938 got 85899345937 for osd.3 2022-03-08T23:09:09.347 INFO:teuthology.orchestra.run.smithi077.stdout:73014444049 2022-03-08T23:09:09.366 INFO:tasks.rook.ceph_manager.ceph:need seq 73014444050 got 73014444049 for osd.0 2022-03-08T23:09:09.444 INFO:teuthology.orchestra.run.smithi077.stdout:90194313230 2022-03-08T23:09:09.466 INFO:tasks.rook.ceph_manager.ceph:need seq 90194313231 got 90194313230 for osd.6 2022-03-08T23:09:09.539 INFO:teuthology.orchestra.run.smithi077.stdout:81604378639 2022-03-08T23:09:09.557 INFO:tasks.rook.ceph_manager.ceph:need seq 81604378640 got 81604378639 for osd.4 2022-03-08T23:09:09.561 INFO:teuthology.orchestra.run.smithi077.stdout:133143986188 2022-03-08T23:09:09.572 INFO:teuthology.orchestra.run.smithi077.stdout:98784247822 2022-03-08T23:09:09.578 INFO:tasks.rook.ceph_manager.ceph:need seq 133143986189 got 133143986188 for osd.10 2022-03-08T23:09:09.591 INFO:tasks.rook.ceph_manager.ceph:need seq 98784247823 got 98784247822 for osd.8 2022-03-08T23:09:09.784 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph osd last-stat-seq osd.2 2022-03-08T23:09:10.052 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph osd last-stat-seq osd.7 2022-03-08T23:09:10.154 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph osd last-stat-seq osd.1 2022-03-08T23:09:10.181 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph osd last-stat-seq osd.5 2022-03-08T23:09:10.214 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph osd last-stat-seq osd.11 2022-03-08T23:09:10.304 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph osd last-stat-seq osd.9 2022-03-08T23:09:10.309 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph osd last-stat-seq osd.3 2022-03-08T23:09:10.322 INFO:teuthology.orchestra.run.smithi077.stdout:94489280527 2022-03-08T23:09:10.367 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph osd last-stat-seq osd.0 2022-03-08T23:09:10.380 INFO:tasks.rook.ceph_manager.ceph:need seq 94489280527 got 94489280527 for osd.2 2022-03-08T23:09:10.380 DEBUG:teuthology.parallel:result is None 2022-03-08T23:09:10.468 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph osd last-stat-seq osd.6 2022-03-08T23:09:10.558 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph osd last-stat-seq osd.4 2022-03-08T23:09:10.579 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph osd last-stat-seq osd.10 2022-03-08T23:09:10.591 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph osd last-stat-seq osd.8 2022-03-08T23:09:10.669 INFO:teuthology.orchestra.run.smithi077.stdout:94489280525 2022-03-08T23:09:10.828 INFO:tasks.rook.ceph_manager.ceph:need seq 94489280525 got 94489280525 for osd.7 2022-03-08T23:09:10.828 DEBUG:teuthology.parallel:result is None 2022-03-08T23:09:10.835 INFO:teuthology.orchestra.run.smithi077.stdout:90194313233 2022-03-08T23:09:10.964 INFO:teuthology.orchestra.run.smithi077.stdout:120259084301 2022-03-08T23:09:11.029 INFO:tasks.rook.ceph_manager.ceph:need seq 90194313232 got 90194313233 for osd.1 2022-03-08T23:09:11.029 INFO:teuthology.orchestra.run.smithi077.stdout:150323855365 2022-03-08T23:09:11.030 DEBUG:teuthology.parallel:result is None 2022-03-08T23:09:11.052 INFO:teuthology.orchestra.run.smithi077.stdout:85899345938 2022-03-08T23:09:11.085 INFO:tasks.rook.ceph_manager.ceph:need seq 120259084301 got 120259084301 for osd.5 2022-03-08T23:09:11.086 DEBUG:teuthology.parallel:result is None 2022-03-08T23:09:11.089 INFO:tasks.rook.ceph_manager.ceph:need seq 150323855365 got 150323855365 for osd.11 2022-03-08T23:09:11.089 DEBUG:teuthology.parallel:result is None 2022-03-08T23:09:11.092 INFO:tasks.rook.ceph_manager.ceph:need seq 85899345938 got 85899345938 for osd.3 2022-03-08T23:09:11.092 DEBUG:teuthology.parallel:result is None 2022-03-08T23:09:11.092 INFO:teuthology.orchestra.run.smithi077.stdout:128849018892 2022-03-08T23:09:11.115 INFO:tasks.rook.ceph_manager.ceph:need seq 128849018892 got 128849018892 for osd.9 2022-03-08T23:09:11.116 DEBUG:teuthology.parallel:result is None 2022-03-08T23:09:11.201 INFO:teuthology.orchestra.run.smithi077.stdout:73014444050 2022-03-08T23:09:11.225 INFO:tasks.rook.ceph_manager.ceph:need seq 73014444050 got 73014444050 for osd.0 2022-03-08T23:09:11.226 DEBUG:teuthology.parallel:result is None 2022-03-08T23:09:11.351 INFO:teuthology.orchestra.run.smithi077.stdout:90194313232 2022-03-08T23:09:11.372 INFO:tasks.rook.ceph_manager.ceph:need seq 90194313231 got 90194313232 for osd.6 2022-03-08T23:09:11.372 DEBUG:teuthology.parallel:result is None 2022-03-08T23:09:11.401 INFO:teuthology.orchestra.run.smithi077.stdout:98784247823 2022-03-08T23:09:11.420 INFO:tasks.rook.ceph_manager.ceph:need seq 98784247823 got 98784247823 for osd.8 2022-03-08T23:09:11.420 DEBUG:teuthology.parallel:result is None 2022-03-08T23:09:11.421 INFO:teuthology.orchestra.run.smithi077.stdout:81604378640 2022-03-08T23:09:11.439 INFO:teuthology.orchestra.run.smithi077.stdout:133143986189 2022-03-08T23:09:11.444 INFO:tasks.rook.ceph_manager.ceph:need seq 81604378640 got 81604378640 for osd.4 2022-03-08T23:09:11.444 DEBUG:teuthology.parallel:result is None 2022-03-08T23:09:11.458 INFO:tasks.rook.ceph_manager.ceph:need seq 133143986189 got 133143986189 for osd.10 2022-03-08T23:09:11.458 DEBUG:teuthology.parallel:result is None 2022-03-08T23:09:11.459 INFO:tasks.rook.ceph_manager.ceph:waiting for clean 2022-03-08T23:09:11.459 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph pg dump --format=json 2022-03-08T23:09:11.928 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-03-08T23:09:11.933 INFO:teuthology.orchestra.run.smithi077.stderr:dumped all 2022-03-08T23:09:11.948 INFO:teuthology.orchestra.run.smithi077.stdout:{"pg_ready":true,"pg_map":{"version":106,"stamp":"2022-03-08T23:09:11.616967+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":12,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":1124941824,"kb_used":66520,"kb_used_data":5880,"kb_used_omap":0,"kb_used_meta":60544,"kb_avail":1124875304,"statfs":{"total":1151940427776,"available":1151872311296,"internally_reserved":0,"allocated":6021120,"data_stored":3564288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":61997056},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003711"},"pg_stats":[{"pgid":"1.0","version":"32'137","reported_seq":195,"reported_epoch":35,"state":"active+clean","last_fresh":"2022-03-08T23:08:55.396844+0000","last_change":"2022-03-08T23:08:10.717038+0000","last_active":"2022-03-08T23:08:55.396844+0000","last_peered":"2022-03-08T23:08:55.396844+0000","last_clean":"2022-03-08T23:08:55.396844+0000","last_became_active":"2022-03-08T23:08:10.716234+0000","last_became_peered":"2022-03-08T23:08:10.716234+0000","last_unstale":"2022-03-08T23:08:55.396844+0000","last_undegraded":"2022-03-08T23:08:55.396844+0000","last_fullsized":"2022-03-08T23:08:55.396844+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-08T23:08:08.705192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-08T23:08:08.705192+0000","last_clean_scrub_stamp":"2022-03-08T23:08:08.705192+0000","objects_scrubbed":0,"log_size":137,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-10T06:17:27.862870+0000","stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":11,"up_from":35,"seq":150323855366,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5168,"kb_used_data":360,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":93739984,"statfs":{"total":95995035648,"available":95989743616,"internally_reserved":0,"allocated":368640,"data_stored":164792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.101}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.202}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.075}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7330000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6970000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5509999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6279999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.573}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6619999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.526}]}]},{"osd":10,"up_from":31,"seq":133143986190,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5232,"kb_used_data":360,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93739920,"statfs":{"total":95995035648,"available":95989678080,"internally_reserved":0,"allocated":368640,"data_stored":164792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34699999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73999999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.52}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2889999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.478}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80500000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5469999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.597}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2110000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98099999999999998}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.244}]}]},{"osd":9,"up_from":30,"seq":128849018893,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5232,"kb_used_data":360,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93739920,"statfs":{"total":95995035648,"available":95989678080,"internally_reserved":0,"allocated":368640,"data_stored":164792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.6840000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.857}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1509999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.2699999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.4050000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.7080000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2200000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.2089999999999996}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.9619999999999997}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.4050000000000002}]}]},{"osd":5,"up_from":28,"seq":120259084301,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5232,"kb_used_data":360,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93739920,"statfs":{"total":95995035648,"available":95989678080,"internally_reserved":0,"allocated":368640,"data_stored":164792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98199999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93000000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.048}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3149999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3839999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7949999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.48}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1319999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8620000000000001}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.833}]}]},{"osd":8,"up_from":23,"seq":98784247824,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5296,"kb_used_data":360,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739856,"statfs":{"total":95995035648,"available":95989612544,"internally_reserved":0,"allocated":368640,"data_stored":164792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3240000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3490000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.5470000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9159999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4370000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4530000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.4820000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.613}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3700000000000001}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.034}]}]},{"osd":7,"up_from":22,"seq":94489280526,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6392,"kb_used_data":880,"kb_used_omap":0,"kb_used_meta":5504,"kb_avail":93738760,"statfs":{"total":95995035648,"available":95988490240,"internally_reserved":0,"allocated":901120,"data_stored":693720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5636096},"hb_peers":[0,1,2,3,4,5,6,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.244}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.036}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.462}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7210000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.083}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3140000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2589999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.7959999999999998}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8380000000000001}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.944}]}]},{"osd":4,"up_from":19,"seq":81604378641,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5296,"kb_used_data":360,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739856,"statfs":{"total":95995035648,"available":95989612544,"internally_reserved":0,"allocated":368640,"data_stored":164792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85099999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0780000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3009999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.581}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0870000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0379999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0030000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.619}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.95}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1600000000000001}]}]},{"osd":0,"up_from":17,"seq":73014444051,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5296,"kb_used_data":360,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739856,"statfs":{"total":95995035648,"available":95989612544,"internally_reserved":0,"allocated":368640,"data_stored":164792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84499999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97499999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0099999999999998}]},{"osd":4,"last update":"Tue Mar 8 23:09:04 2022","interfaces":[{"interface":"back","average":{"1min":1.8089999999999999,"5min":1.8089999999999999,"15min":1.8089999999999999},"min":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"max":{"1min":2.298,"5min":2.298,"15min":2.298},"last":1.361},{"interface":"front","average":{"1min":1.5329999999999999,"5min":1.5329999999999999,"15min":1.5329999999999999},"min":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"max":{"1min":2.3359999999999999,"5min":2.3359999999999999,"15min":2.3359999999999999},"last":1.5029999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4259999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5569999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.706}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6759999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.155}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0720000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7889999999999999}]}]},{"osd":3,"up_from":20,"seq":85899345938,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6392,"kb_used_data":880,"kb_used_omap":0,"kb_used_meta":5504,"kb_avail":93738760,"statfs":{"total":95995035648,"available":95988490240,"internally_reserved":0,"allocated":901120,"data_stored":693720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5636096},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Mar 8 23:09:06 2022","interfaces":[{"interface":"back","average":{"1min":1.163,"5min":1.163,"15min":1.163},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.8740000000000001,"5min":1.8740000000000001,"15min":1.8740000000000001},"last":1.359},{"interface":"front","average":{"1min":1.0940000000000001,"5min":1.0940000000000001,"15min":1.0940000000000001},"min":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":1.6180000000000001,"5min":1.6180000000000001,"15min":1.6180000000000001},"last":1.611}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.444}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.4630000000000001}]},{"osd":4,"last update":"Tue Mar 8 23:09:06 2022","interfaces":[{"interface":"back","average":{"1min":1.474,"5min":1.474,"15min":1.474},"min":{"1min":0.745,"5min":0.745,"15min":0.745},"max":{"1min":2.3130000000000002,"5min":2.3130000000000002,"15min":2.3130000000000002},"last":2.3130000000000002},{"interface":"front","average":{"1min":1.4470000000000001,"5min":1.4470000000000001,"15min":1.4470000000000001},"min":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"max":{"1min":2.0659999999999998,"5min":2.0659999999999998,"15min":2.0659999999999998},"last":1.4990000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.677}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.825}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7649999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.069}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.351}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3999999999999999}]}]},{"osd":1,"up_from":21,"seq":90194313233,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5296,"kb_used_data":360,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739856,"statfs":{"total":95995035648,"available":95989612544,"internally_reserved":0,"allocated":368640,"data_stored":164792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51300000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5189999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.554}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76900000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4730000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.962}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0750000000000002}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0289999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1659999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.081}]}]},{"osd":6,"up_from":21,"seq":90194313232,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5296,"kb_used_data":360,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739856,"statfs":{"total":95995035648,"available":95989612544,"internally_reserved":0,"allocated":368640,"data_stored":164792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.238}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4950000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.337}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92000000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0920000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8919999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2949999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1680000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1259999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.476}]}]},{"osd":2,"up_from":22,"seq":94489280528,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6392,"kb_used_data":880,"kb_used_omap":0,"kb_used_meta":5504,"kb_avail":93738760,"statfs":{"total":95995035648,"available":95988490240,"internally_reserved":0,"allocated":901120,"data_stored":693720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5636096},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.526}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9139999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3049999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9359999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.248}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.278}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.0490000000000004}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3700000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.0899999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.008}]}]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2022-03-08T23:09:11.950 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph pg dump --format=json 2022-03-08T23:09:12.424 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-03-08T23:09:12.427 INFO:teuthology.orchestra.run.smithi077.stderr:dumped all 2022-03-08T23:09:12.447 INFO:teuthology.orchestra.run.smithi077.stdout:{"pg_ready":true,"pg_map":{"version":106,"stamp":"2022-03-08T23:09:11.616967+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":12,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":1124941824,"kb_used":66520,"kb_used_data":5880,"kb_used_omap":0,"kb_used_meta":60544,"kb_avail":1124875304,"statfs":{"total":1151940427776,"available":1151872311296,"internally_reserved":0,"allocated":6021120,"data_stored":3564288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":61997056},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003711"},"pg_stats":[{"pgid":"1.0","version":"32'137","reported_seq":195,"reported_epoch":35,"state":"active+clean","last_fresh":"2022-03-08T23:08:55.396844+0000","last_change":"2022-03-08T23:08:10.717038+0000","last_active":"2022-03-08T23:08:55.396844+0000","last_peered":"2022-03-08T23:08:55.396844+0000","last_clean":"2022-03-08T23:08:55.396844+0000","last_became_active":"2022-03-08T23:08:10.716234+0000","last_became_peered":"2022-03-08T23:08:10.716234+0000","last_unstale":"2022-03-08T23:08:55.396844+0000","last_undegraded":"2022-03-08T23:08:55.396844+0000","last_fullsized":"2022-03-08T23:08:55.396844+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-08T23:08:08.705192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-08T23:08:08.705192+0000","last_clean_scrub_stamp":"2022-03-08T23:08:08.705192+0000","objects_scrubbed":0,"log_size":137,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-10T06:17:27.862870+0000","stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":11,"up_from":35,"seq":150323855366,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5168,"kb_used_data":360,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":93739984,"statfs":{"total":95995035648,"available":95989743616,"internally_reserved":0,"allocated":368640,"data_stored":164792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.101}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.202}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.075}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7330000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6970000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5509999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6279999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.573}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6619999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.526}]}]},{"osd":10,"up_from":31,"seq":133143986190,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5232,"kb_used_data":360,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93739920,"statfs":{"total":95995035648,"available":95989678080,"internally_reserved":0,"allocated":368640,"data_stored":164792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34699999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73999999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.52}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2889999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.478}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80500000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5469999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.597}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2110000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98099999999999998}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.244}]}]},{"osd":9,"up_from":30,"seq":128849018893,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5232,"kb_used_data":360,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93739920,"statfs":{"total":95995035648,"available":95989678080,"internally_reserved":0,"allocated":368640,"data_stored":164792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.6840000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.857}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1509999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.2699999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.4050000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.7080000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2200000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.2089999999999996}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.9619999999999997}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.4050000000000002}]}]},{"osd":5,"up_from":28,"seq":120259084301,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5232,"kb_used_data":360,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93739920,"statfs":{"total":95995035648,"available":95989678080,"internally_reserved":0,"allocated":368640,"data_stored":164792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98199999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93000000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.048}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3149999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3839999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7949999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.48}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1319999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8620000000000001}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.833}]}]},{"osd":8,"up_from":23,"seq":98784247824,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5296,"kb_used_data":360,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739856,"statfs":{"total":95995035648,"available":95989612544,"internally_reserved":0,"allocated":368640,"data_stored":164792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3240000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3490000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.5470000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9159999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4370000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4530000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.4820000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.613}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3700000000000001}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.034}]}]},{"osd":7,"up_from":22,"seq":94489280526,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6392,"kb_used_data":880,"kb_used_omap":0,"kb_used_meta":5504,"kb_avail":93738760,"statfs":{"total":95995035648,"available":95988490240,"internally_reserved":0,"allocated":901120,"data_stored":693720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5636096},"hb_peers":[0,1,2,3,4,5,6,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.244}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.036}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.462}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7210000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.083}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3140000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2589999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.7959999999999998}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8380000000000001}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.944}]}]},{"osd":4,"up_from":19,"seq":81604378641,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5296,"kb_used_data":360,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739856,"statfs":{"total":95995035648,"available":95989612544,"internally_reserved":0,"allocated":368640,"data_stored":164792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85099999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0780000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3009999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.581}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0870000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0379999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0030000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.619}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.95}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1600000000000001}]}]},{"osd":0,"up_from":17,"seq":73014444051,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5296,"kb_used_data":360,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739856,"statfs":{"total":95995035648,"available":95989612544,"internally_reserved":0,"allocated":368640,"data_stored":164792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84499999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97499999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0099999999999998}]},{"osd":4,"last update":"Tue Mar 8 23:09:04 2022","interfaces":[{"interface":"back","average":{"1min":1.8089999999999999,"5min":1.8089999999999999,"15min":1.8089999999999999},"min":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"max":{"1min":2.298,"5min":2.298,"15min":2.298},"last":1.361},{"interface":"front","average":{"1min":1.5329999999999999,"5min":1.5329999999999999,"15min":1.5329999999999999},"min":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"max":{"1min":2.3359999999999999,"5min":2.3359999999999999,"15min":2.3359999999999999},"last":1.5029999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4259999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5569999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.706}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6759999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.155}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0720000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7889999999999999}]}]},{"osd":3,"up_from":20,"seq":85899345938,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6392,"kb_used_data":880,"kb_used_omap":0,"kb_used_meta":5504,"kb_avail":93738760,"statfs":{"total":95995035648,"available":95988490240,"internally_reserved":0,"allocated":901120,"data_stored":693720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5636096},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Mar 8 23:09:06 2022","interfaces":[{"interface":"back","average":{"1min":1.163,"5min":1.163,"15min":1.163},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.8740000000000001,"5min":1.8740000000000001,"15min":1.8740000000000001},"last":1.359},{"interface":"front","average":{"1min":1.0940000000000001,"5min":1.0940000000000001,"15min":1.0940000000000001},"min":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":1.6180000000000001,"5min":1.6180000000000001,"15min":1.6180000000000001},"last":1.611}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.444}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.4630000000000001}]},{"osd":4,"last update":"Tue Mar 8 23:09:06 2022","interfaces":[{"interface":"back","average":{"1min":1.474,"5min":1.474,"15min":1.474},"min":{"1min":0.745,"5min":0.745,"15min":0.745},"max":{"1min":2.3130000000000002,"5min":2.3130000000000002,"15min":2.3130000000000002},"last":2.3130000000000002},{"interface":"front","average":{"1min":1.4470000000000001,"5min":1.4470000000000001,"15min":1.4470000000000001},"min":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"max":{"1min":2.0659999999999998,"5min":2.0659999999999998,"15min":2.0659999999999998},"last":1.4990000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.677}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.825}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7649999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.069}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.351}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3999999999999999}]}]},{"osd":1,"up_from":21,"seq":90194313233,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5296,"kb_used_data":360,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739856,"statfs":{"total":95995035648,"available":95989612544,"internally_reserved":0,"allocated":368640,"data_stored":164792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51300000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5189999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.554}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76900000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4730000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.962}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0750000000000002}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0289999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1659999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.081}]}]},{"osd":6,"up_from":21,"seq":90194313232,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5296,"kb_used_data":360,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739856,"statfs":{"total":95995035648,"available":95989612544,"internally_reserved":0,"allocated":368640,"data_stored":164792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.238}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4950000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.337}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92000000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0920000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8919999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2949999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1680000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1259999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.476}]}]},{"osd":2,"up_from":22,"seq":94489280528,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6392,"kb_used_data":880,"kb_used_omap":0,"kb_used_meta":5504,"kb_avail":93738760,"statfs":{"total":95995035648,"available":95988490240,"internally_reserved":0,"allocated":901120,"data_stored":693720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5636096},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.526}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9139999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3049999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9359999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.248}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.278}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.0490000000000004}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3700000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.0899999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.008}]}]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2022-03-08T23:09:12.449 INFO:tasks.rook.ceph_manager.ceph:clean! 2022-03-08T23:09:12.449 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2022-03-08T23:09:12.450 INFO:tasks.rook.ceph_manager.ceph:wait_until_healthy 2022-03-08T23:09:12.450 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph health --format=json 2022-03-08T23:09:12.976 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-03-08T23:09:12.976 INFO:teuthology.orchestra.run.smithi077.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2022-03-08T23:09:13.004 INFO:tasks.rook.ceph_manager.ceph:wait_until_healthy done 2022-03-08T23:09:13.005 INFO:tasks.rook:Rook complete, yielding 2022-03-08T23:09:13.005 INFO:teuthology.run_tasks:Running task rook.shell... 2022-03-08T23:09:13.016 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph -s 2022-03-08T23:09:13.617 INFO:teuthology.orchestra.run.smithi077.stdout: cluster: 2022-03-08T23:09:13.618 INFO:teuthology.orchestra.run.smithi077.stdout: id: 965c96f5-6497-4297-a748-9f1a68fefb7b 2022-03-08T23:09:13.618 INFO:teuthology.orchestra.run.smithi077.stdout: health: HEALTH_OK 2022-03-08T23:09:13.618 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-03-08T23:09:13.618 INFO:teuthology.orchestra.run.smithi077.stdout: services: 2022-03-08T23:09:13.619 INFO:teuthology.orchestra.run.smithi077.stdout: mon: 3 daemons, quorum a,b,c (age 3m) 2022-03-08T23:09:13.619 INFO:teuthology.orchestra.run.smithi077.stdout: mgr: a(active, since 2m) 2022-03-08T23:09:13.619 INFO:teuthology.orchestra.run.smithi077.stdout: osd: 12 osds: 12 up (since 18s), 12 in (since 94s) 2022-03-08T23:09:13.620 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-03-08T23:09:13.620 INFO:teuthology.orchestra.run.smithi077.stdout: data: 2022-03-08T23:09:13.620 INFO:teuthology.orchestra.run.smithi077.stdout: pools: 1 pools, 1 pgs 2022-03-08T23:09:13.620 INFO:teuthology.orchestra.run.smithi077.stdout: objects: 2 objects, 577 KiB 2022-03-08T23:09:13.621 INFO:teuthology.orchestra.run.smithi077.stdout: usage: 65 MiB used, 1.0 TiB / 1.0 TiB avail 2022-03-08T23:09:13.621 INFO:teuthology.orchestra.run.smithi077.stdout: pgs: 1 active+clean 2022-03-08T23:09:13.621 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-03-08T23:09:13.637 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph orch status 2022-03-08T23:09:14.360 INFO:teuthology.orchestra.run.smithi077.stdout:Backend: rook 2022-03-08T23:09:14.360 INFO:teuthology.orchestra.run.smithi077.stdout:Available: Yes 2022-03-08T23:09:14.384 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph orch ps 2022-03-08T23:09:14.971 INFO:teuthology.orchestra.run.smithi077.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID 2022-03-08T23:09:14.971 INFO:teuthology.orchestra.run.smithi077.stdout:crashcollector.crash smithi077 running (2m) 0s ago 2m - - 4b871cde5e1a 2022-03-08T23:09:14.971 INFO:teuthology.orchestra.run.smithi077.stdout:crashcollector.crash smithi106 running (105s) 0s ago 105s - - 4b871cde5e1a 2022-03-08T23:09:14.972 INFO:teuthology.orchestra.run.smithi077.stdout:crashcollector.crash smithi135 running (2m) 0s ago 2m - - 4b871cde5e1a 2022-03-08T23:09:14.972 INFO:teuthology.orchestra.run.smithi077.stdout:mgr.a smithi106 running (3m) 0s ago 3m - - 4b871cde5e1a 2022-03-08T23:09:14.972 INFO:teuthology.orchestra.run.smithi077.stdout:mon.a smithi135 running (5m) 0s ago 5m - - 4b871cde5e1a 2022-03-08T23:09:14.972 INFO:teuthology.orchestra.run.smithi077.stdout:mon.b smithi106 running (5m) 0s ago 5m - - 4b871cde5e1a 2022-03-08T23:09:14.973 INFO:teuthology.orchestra.run.smithi077.stdout:mon.c smithi077 running (3m) 0s ago 3m - - 4b871cde5e1a 2022-03-08T23:09:14.973 INFO:teuthology.orchestra.run.smithi077.stdout:osd.0 smithi077 running (108s) 0s ago 108s - - 4b871cde5e1a 2022-03-08T23:09:14.973 INFO:teuthology.orchestra.run.smithi077.stdout:osd.1 smithi106 running (106s) 0s ago 107s - - 4b871cde5e1a 2022-03-08T23:09:14.973 INFO:teuthology.orchestra.run.smithi077.stdout:osd.10 smithi077 running (89s) 0s ago 89s - - 4b871cde5e1a 2022-03-08T23:09:14.973 INFO:teuthology.orchestra.run.smithi077.stdout:osd.11 smithi106 running (41s) 0s ago 41s - - 4b871cde5e1a 2022-03-08T23:09:14.974 INFO:teuthology.orchestra.run.smithi077.stdout:osd.2 smithi106 running (104s) 0s ago 104s - - 4b871cde5e1a 2022-03-08T23:09:14.974 INFO:teuthology.orchestra.run.smithi077.stdout:osd.3 smithi135 running (104s) 0s ago 104s - - 4b871cde5e1a 2022-03-08T23:09:14.974 INFO:teuthology.orchestra.run.smithi077.stdout:osd.4 smithi135 running (103s) 0s ago 103s - - 4b871cde5e1a 2022-03-08T23:09:14.974 INFO:teuthology.orchestra.run.smithi077.stdout:osd.5 smithi077 running (99s) 0s ago 99s - - 4b871cde5e1a 2022-03-08T23:09:14.975 INFO:teuthology.orchestra.run.smithi077.stdout:osd.6 smithi135 running (95s) 0s ago 96s - - 4b871cde5e1a 2022-03-08T23:09:14.975 INFO:teuthology.orchestra.run.smithi077.stdout:osd.7 smithi135 running (95s) 0s ago 95s - - 4b871cde5e1a 2022-03-08T23:09:14.975 INFO:teuthology.orchestra.run.smithi077.stdout:osd.8 smithi106 running (93s) 0s ago 94s - - 4b871cde5e1a 2022-03-08T23:09:14.975 INFO:teuthology.orchestra.run.smithi077.stdout:osd.9 smithi077 running (91s) 0s ago 92s - - 4b871cde5e1a 2022-03-08T23:09:14.993 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph orch ls 2022-03-08T23:09:15.652 INFO:teuthology.orchestra.run.smithi077.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2022-03-08T23:09:15.652 INFO:teuthology.orchestra.run.smithi077.stdout:crash 3/3 0s ago 2m * 2022-03-08T23:09:15.653 INFO:teuthology.orchestra.run.smithi077.stdout:mgr 1/1 0s ago 3m count:1 2022-03-08T23:09:15.653 INFO:teuthology.orchestra.run.smithi077.stdout:mon 3/3 0s ago 5m count:3 2022-03-08T23:09:15.653 INFO:teuthology.orchestra.run.smithi077.stdout:osd 24 0s ago 109s 2022-03-08T23:09:15.653 INFO:teuthology.orchestra.run.smithi077.stdout:osd.all-available-devices 0 0s ago - * 2022-03-08T23:09:15.674 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph orch device ls 2022-03-08T23:09:16.224 INFO:teuthology.orchestra.run.smithi077.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2022-03-08T23:09:16.224 INFO:teuthology.orchestra.run.smithi077.stdout:smithi077 /dev/vg_nvme/lv_1 unknown None 107G 0s ago 2022-03-08T23:09:16.224 INFO:teuthology.orchestra.run.smithi077.stdout:smithi077 /dev/vg_nvme/lv_2 unknown None 107G 0s ago 2022-03-08T23:09:16.225 INFO:teuthology.orchestra.run.smithi077.stdout:smithi077 /dev/vg_nvme/lv_3 unknown None 107G 0s ago 2022-03-08T23:09:16.225 INFO:teuthology.orchestra.run.smithi077.stdout:smithi077 /dev/vg_nvme/lv_4 unknown None 107G 0s ago 2022-03-08T23:09:16.225 INFO:teuthology.orchestra.run.smithi077.stdout:smithi106 /dev/vg_nvme/lv_1 unknown None 107G 0s ago 2022-03-08T23:09:16.225 INFO:teuthology.orchestra.run.smithi077.stdout:smithi106 /dev/vg_nvme/lv_2 unknown None 107G 0s ago 2022-03-08T23:09:16.225 INFO:teuthology.orchestra.run.smithi077.stdout:smithi106 /dev/vg_nvme/lv_3 unknown None 107G 0s ago 2022-03-08T23:09:16.226 INFO:teuthology.orchestra.run.smithi077.stdout:smithi106 /dev/vg_nvme/lv_4 unknown None 107G 0s ago 2022-03-08T23:09:16.226 INFO:teuthology.orchestra.run.smithi077.stdout:smithi135 /dev/vg_nvme/lv_1 unknown None 107G 0s ago 2022-03-08T23:09:16.226 INFO:teuthology.orchestra.run.smithi077.stdout:smithi135 /dev/vg_nvme/lv_2 unknown None 107G 0s ago 2022-03-08T23:09:16.226 INFO:teuthology.orchestra.run.smithi077.stdout:smithi135 /dev/vg_nvme/lv_3 unknown None 107G 0s ago 2022-03-08T23:09:16.226 INFO:teuthology.orchestra.run.smithi077.stdout:smithi135 /dev/vg_nvme/lv_4 unknown None 107G 0s ago 2022-03-08T23:09:16.248 INFO:teuthology.run_tasks:Running task exec... 2022-03-08T23:09:16.262 INFO:teuthology.task.exec:Executing custom commands... 2022-03-08T23:09:16.262 INFO:teuthology.task.exec:Running commands on role host.a host ubuntu@smithi077.front.sepia.ceph.com 2022-03-08T23:09:16.263 DEBUG:teuthology.orchestra.run.smithi077:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'set -ex 2022-03-08T23:09:16.263 DEBUG:teuthology.orchestra.run.smithi077:> toolbox() { 2022-03-08T23:09:16.263 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- "$@" 2022-03-08T23:09:16.264 DEBUG:teuthology.orchestra.run.smithi077:> } 2022-03-08T23:09:16.264 DEBUG:teuthology.orchestra.run.smithi077:> orig_num_osd=`toolbox ceph osd stat | cut -f3 -d " "` 2022-03-08T23:09:16.264 DEBUG:teuthology.orchestra.run.smithi077:> toolbox ceph orch osd rm 0 --force 2022-03-08T23:09:16.264 DEBUG:teuthology.orchestra.run.smithi077:> removed_pv="" 2022-03-08T23:09:16.265 DEBUG:teuthology.orchestra.run.smithi077:> while [ "$removed_pv" = "" ] 2022-03-08T23:09:16.265 DEBUG:teuthology.orchestra.run.smithi077:> do 2022-03-08T23:09:16.265 DEBUG:teuthology.orchestra.run.smithi077:> removed_pv=`kubectl get pv | grep Released | cut -f1 -d " "` 2022-03-08T23:09:16.265 DEBUG:teuthology.orchestra.run.smithi077:> sleep 3s 2022-03-08T23:09:16.266 DEBUG:teuthology.orchestra.run.smithi077:> done 2022-03-08T23:09:16.266 DEBUG:teuthology.orchestra.run.smithi077:> target_path=`kubectl get pv $removed_pv -o jsonpath='"'"'{.spec.local.path}'"'"'` 2022-03-08T23:09:16.266 DEBUG:teuthology.orchestra.run.smithi077:> host=`echo $removed_pv | cut -f1 -d "-"` 2022-03-08T23:09:16.267 DEBUG:teuthology.orchestra.run.smithi077:> toolbox ceph orch device zap $host $target_path --force 2022-03-08T23:09:16.267 DEBUG:teuthology.orchestra.run.smithi077:> zap_completion="0" 2022-03-08T23:09:16.267 DEBUG:teuthology.orchestra.run.smithi077:> while [ "$zap_completion" = "0" ] 2022-03-08T23:09:16.267 DEBUG:teuthology.orchestra.run.smithi077:> do 2022-03-08T23:09:16.268 DEBUG:teuthology.orchestra.run.smithi077:> zap_completion=`kubectl get job -n rook-ceph rook-ceph-device-zap -o jsonpath='"'"'{.status.succeeded.path}'"'"'` 2022-03-08T23:09:16.268 DEBUG:teuthology.orchestra.run.smithi077:> sleep 3s 2022-03-08T23:09:16.268 DEBUG:teuthology.orchestra.run.smithi077:> done 2022-03-08T23:09:16.269 DEBUG:teuthology.orchestra.run.smithi077:> kubectl patch pv $removed_pv -p '"'"'{"spec":{"claimRef": null}}'"'"' 2022-03-08T23:09:16.269 DEBUG:teuthology.orchestra.run.smithi077:> toolbox ceph orch apply osd --all-available-devices 2022-03-08T23:09:16.269 DEBUG:teuthology.orchestra.run.smithi077:> kubectl delete job rook-ceph-device-zap -n rook-ceph 2022-03-08T23:09:16.269 DEBUG:teuthology.orchestra.run.smithi077:> num_osd="0" 2022-03-08T23:09:16.270 DEBUG:teuthology.orchestra.run.smithi077:> while [ "$num_osd" != "$orig_num_osd" ] 2022-03-08T23:09:16.270 DEBUG:teuthology.orchestra.run.smithi077:> do 2022-03-08T23:09:16.270 DEBUG:teuthology.orchestra.run.smithi077:> echo "waiting for osd to come back up" 2022-03-08T23:09:16.271 DEBUG:teuthology.orchestra.run.smithi077:> num_osd=`toolbox ceph osd stat | cut -f3 -d " "` 2022-03-08T23:09:16.271 DEBUG:teuthology.orchestra.run.smithi077:> sleep 30s 2022-03-08T23:09:16.271 DEBUG:teuthology.orchestra.run.smithi077:> done 2022-03-08T23:09:16.271 DEBUG:teuthology.orchestra.run.smithi077:> ' 2022-03-08T23:09:16.289 INFO:teuthology.orchestra.run.smithi077.stderr:++ toolbox ceph osd stat 2022-03-08T23:09:16.289 INFO:teuthology.orchestra.run.smithi077.stderr:++ cut -f3 -d ' ' 2022-03-08T23:09:16.289 INFO:teuthology.orchestra.run.smithi077.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-03-08T23:09:18.454 INFO:teuthology.orchestra.run.smithi077.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-08T23:09:18.940 INFO:teuthology.orchestra.run.smithi077.stderr:+ orig_num_osd=12 2022-03-08T23:09:18.941 INFO:teuthology.orchestra.run.smithi077.stderr:+ toolbox ceph orch osd rm 0 --force 2022-03-08T23:09:18.941 INFO:teuthology.orchestra.run.smithi077.stderr:+ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph orch osd rm 0 --force 2022-03-08T23:09:19.019 INFO:teuthology.orchestra.run.smithi077.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-08T23:09:19.655 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:19.652714 I | ceph-cluster-controller: CR has changed for "rook-ceph". diff=  v1.ClusterSpec{ 2022-03-08T23:09:19.656 INFO:tasks.rook.operator.smithi077.stdout:   CephVersion: {Image: "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", AllowUnsupported: true}, 2022-03-08T23:09:19.656 INFO:tasks.rook.operator.smithi077.stdout:   Storage: v1.StorageScopeSpec{ 2022-03-08T23:09:19.656 INFO:tasks.rook.operator.smithi077.stdout:   ... // 3 identical fields 2022-03-08T23:09:19.657 INFO:tasks.rook.operator.smithi077.stdout:   Config: nil, 2022-03-08T23:09:19.657 INFO:tasks.rook.operator.smithi077.stdout:   Selection: {}, 2022-03-08T23:09:19.657 INFO:tasks.rook.operator.smithi077.stdout:   StorageClassDeviceSets: []v1.StorageClassDeviceSet{ 2022-03-08T23:09:19.657 INFO:tasks.rook.operator.smithi077.stdout:   {Name: "smithi077-lv-1", Count: 1, VolumeClaimTemplates: {{ObjectMeta: {Name: "data"}, Spec: {AccessModes: {"ReadWriteOnce"}, Resources: {Requests: {s"storage": {i: {...}, s: "1", Format: "DecimalSI"}}}, VolumeName: "smithi077-lv-1", StorageClassName: &"scratch", ...}}}}, 2022-03-08T23:09:19.658 INFO:tasks.rook.operator.smithi077.stdout:   {Name: "smithi077-lv-2", Count: 1, VolumeClaimTemplates: {{ObjectMeta: {Name: "data"}, Spec: {AccessModes: {"ReadWriteOnce"}, Resources: {Requests: {s"storage": {i: {...}, s: "1", Format: "DecimalSI"}}}, VolumeName: "smithi077-lv-2", StorageClassName: &"scratch", ...}}}}, 2022-03-08T23:09:19.658 INFO:tasks.rook.operator.smithi077.stdout:-  { 2022-03-08T23:09:19.658 INFO:tasks.rook.operator.smithi077.stdout:-  Name: "smithi077-lv-3", 2022-03-08T23:09:19.658 INFO:tasks.rook.operator.smithi077.stdout:-  Count: 1, 2022-03-08T23:09:19.658 INFO:tasks.rook.operator.smithi077.stdout:-  VolumeClaimTemplates: []v1.PersistentVolumeClaim{{ObjectMeta: v1.ObjectMeta{...}, Spec: v1.PersistentVolumeClaimSpec{...}}}, 2022-03-08T23:09:19.659 INFO:tasks.rook.operator.smithi077.stdout:-  }, 2022-03-08T23:09:19.659 INFO:tasks.rook.operator.smithi077.stdout:   {Name: "smithi077-lv-4", Count: 1, VolumeClaimTemplates: {{ObjectMeta: {Name: "data"}, Spec: {AccessModes: {"ReadWriteOnce"}, Resources: {Requests: {s"storage": {i: {...}, s: "1", Format: "DecimalSI"}}}, VolumeName: "smithi077-lv-4", StorageClassName: &"scratch", ...}}}}, 2022-03-08T23:09:19.659 INFO:tasks.rook.operator.smithi077.stdout:   {Name: "smithi106-lv-1", Count: 1, VolumeClaimTemplates: {{ObjectMeta: {Name: "data"}, Spec: {AccessModes: {"ReadWriteOnce"}, Resources: {Requests: {s"storage": {i: {...}, s: "1", Format: "DecimalSI"}}}, VolumeName: "smithi106-lv-1", StorageClassName: &"scratch", ...}}}}, 2022-03-08T23:09:19.659 INFO:tasks.rook.operator.smithi077.stdout:   ... // 7 identical elements 2022-03-08T23:09:19.660 INFO:tasks.rook.operator.smithi077.stdout:   }, 2022-03-08T23:09:19.660 INFO:tasks.rook.operator.smithi077.stdout:   }, 2022-03-08T23:09:19.660 INFO:tasks.rook.operator.smithi077.stdout:   Annotations: nil, 2022-03-08T23:09:19.660 INFO:tasks.rook.operator.smithi077.stdout:   Labels: nil, 2022-03-08T23:09:19.660 INFO:tasks.rook.operator.smithi077.stdout:   ... // 20 identical fields 2022-03-08T23:09:19.661 INFO:tasks.rook.operator.smithi077.stdout:  } 2022-03-08T23:09:19.661 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:19.652824 I | operator: reloading operator's CRDs manager, cancelling all orchestrations! 2022-03-08T23:09:19.661 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:19.653110 I | operator: successfully started the controller-runtime manager 2022-03-08T23:09:19.661 INFO:tasks.rook.operator.smithi077.stdout:I0308 23:09:19.653322 1 manager.go:148] objectbucket.io/provisioner-manager "msg"="stopping provisioner" "name"="rook-ceph.ceph.rook.io/bucket" "reason"={} 2022-03-08T23:09:19.669 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:19.668276 I | op-k8sutil: ROOK_CURRENT_NAMESPACE_ONLY="false" (env var) 2022-03-08T23:09:19.670 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:19.668296 I | operator: watching all namespaces for Ceph CRs 2022-03-08T23:09:19.670 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:19.668346 I | operator: setting up schemes 2022-03-08T23:09:19.673 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:19.671147 I | operator: setting up the controller-runtime manager 2022-03-08T23:09:19.877 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:19.875222 I | operator: looking for admission webhook secret "rook-ceph-admission-controller" 2022-03-08T23:09:19.940 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:19.936999 I | operator: admission webhook secret "rook-ceph-admission-controller" not found. proceeding without the admission controller 2022-03-08T23:09:19.941 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:19.937073 I | ceph-cluster-controller: successfully started 2022-03-08T23:09:19.941 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:19.937165 I | ceph-cluster-controller: enabling hotplug orchestration 2022-03-08T23:09:19.941 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:19.937205 I | ceph-crashcollector-controller: successfully started 2022-03-08T23:09:19.941 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:19.937244 I | ceph-block-pool-controller: successfully started 2022-03-08T23:09:19.942 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:19.937289 I | ceph-object-store-user-controller: successfully started 2022-03-08T23:09:19.942 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:19.937343 I | ceph-object-realm-controller: successfully started 2022-03-08T23:09:19.942 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:19.937371 I | ceph-object-zonegroup-controller: successfully started 2022-03-08T23:09:19.942 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:19.937396 I | ceph-object-zone-controller: successfully started 2022-03-08T23:09:19.943 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:19.937592 I | ceph-object-controller: successfully started 2022-03-08T23:09:19.943 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:19.937637 I | ceph-file-controller: successfully started 2022-03-08T23:09:19.943 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:19.937689 I | ceph-nfs-controller: successfully started 2022-03-08T23:09:19.944 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:19.937726 I | ceph-rbd-mirror-controller: successfully started 2022-03-08T23:09:19.944 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:19.937767 I | ceph-client-controller: successfully started 2022-03-08T23:09:19.945 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:19.937793 I | ceph-filesystem-mirror-controller: successfully started 2022-03-08T23:09:19.945 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:19.937840 I | operator: rook-ceph-operator-config-controller successfully started 2022-03-08T23:09:19.945 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:19.937865 I | ceph-csi: rook-ceph-operator-csi-controller successfully started 2022-03-08T23:09:19.945 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:19.937886 I | op-bucket-prov: rook-ceph-operator-bucket-controller successfully started 2022-03-08T23:09:19.946 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:19.937918 I | ceph-bucket-topic: successfully started 2022-03-08T23:09:19.946 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:19.937933 I | ceph-bucket-notification: successfully started 2022-03-08T23:09:19.946 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:19.937946 I | ceph-bucket-notification: successfully started 2022-03-08T23:09:19.946 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:19.937959 I | ceph-fs-subvolumegroup-controller: successfully started 2022-03-08T23:09:19.947 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:19.939832 I | operator: starting the controller-runtime manager 2022-03-08T23:09:20.045 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:20.043362 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-03-08T23:09:20.049 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:20.046037 I | op-k8sutil: ROOK_CEPH_COMMANDS_TIMEOUT_SECONDS="15" (configmap) 2022-03-08T23:09:20.049 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:20.046090 I | op-k8sutil: ROOK_LOG_LEVEL="INFO" (configmap) 2022-03-08T23:09:20.050 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:20.046106 I | op-k8sutil: ROOK_ENABLE_DISCOVERY_DAEMON="false" (configmap) 2022-03-08T23:09:20.100 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:20.098202 I | operator: rook-ceph-operator-config-controller done reconciling 2022-03-08T23:09:20.150 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:20.144523 I | op-mon: parsing mon endpoints: a=10.110.141.123:6789,b=10.98.41.230:6789,c=10.109.47.142:6789 2022-03-08T23:09:20.150 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:20.144865 I | op-mon: parsing mon endpoints: a=10.110.141.123:6789,b=10.98.41.230:6789,c=10.109.47.142:6789 2022-03-08T23:09:20.153 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:20.144904 I | op-k8sutil: ROOK_OBC_WATCH_OPERATOR_NAMESPACE="true" (configmap) 2022-03-08T23:09:20.153 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:20.144913 I | op-bucket-prov: ceph bucket provisioner launched watching for provisioner "rook-ceph.ceph.rook.io/bucket" 2022-03-08T23:09:20.153 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:20.146444 I | op-bucket-prov: successfully reconciled bucket provisioner 2022-03-08T23:09:20.154 INFO:tasks.rook.operator.smithi077.stdout:I0308 23:09:20.146510 1 manager.go:135] objectbucket.io/provisioner-manager "msg"="starting provisioner" "name"="rook-ceph.ceph.rook.io/bucket" 2022-03-08T23:09:20.314 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:20.312116 I | ceph-csi: successfully created csi config map "rook-ceph-csi-config" 2022-03-08T23:09:20.320 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:20.318679 I | op-k8sutil: ROOK_CSI_ENABLE_RBD="true" (configmap) 2022-03-08T23:09:20.320 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:20.318698 I | op-k8sutil: ROOK_CSI_ENABLE_CEPHFS="true" (configmap) 2022-03-08T23:09:20.320 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:20.318704 I | op-k8sutil: ROOK_CSI_ALLOW_UNSUPPORTED_VERSION="false" (configmap) 2022-03-08T23:09:20.320 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:20.318708 I | op-k8sutil: ROOK_CSI_ENABLE_GRPC_METRICS="false" (configmap) 2022-03-08T23:09:20.321 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:20.318713 I | op-k8sutil: CSI_ENABLE_HOST_NETWORK="true" (default) 2022-03-08T23:09:20.322 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:20.318718 I | op-k8sutil: ROOK_CSI_CEPH_IMAGE="quay.io/cephcsi/cephcsi:v3.5.1" (default) 2022-03-08T23:09:20.322 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:20.318724 I | op-k8sutil: ROOK_CSI_REGISTRAR_IMAGE="k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.5.0" (default) 2022-03-08T23:09:20.322 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:20.318729 I | op-k8sutil: ROOK_CSI_PROVISIONER_IMAGE="k8s.gcr.io/sig-storage/csi-provisioner:v3.1.0" (default) 2022-03-08T23:09:20.322 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:20.318734 I | op-k8sutil: ROOK_CSI_ATTACHER_IMAGE="k8s.gcr.io/sig-storage/csi-attacher:v3.4.0" (default) 2022-03-08T23:09:20.323 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:20.318740 I | op-k8sutil: ROOK_CSI_SNAPSHOTTER_IMAGE="k8s.gcr.io/sig-storage/csi-snapshotter:v5.0.1" (default) 2022-03-08T23:09:20.323 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:20.318744 I | op-k8sutil: ROOK_CSI_KUBELET_DIR_PATH="/var/lib/kubelet" (default) 2022-03-08T23:09:20.323 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:20.318750 I | op-k8sutil: CSI_VOLUME_REPLICATION_IMAGE="quay.io/csiaddons/volumereplication-operator:v0.3.0" (default) 2022-03-08T23:09:20.323 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:20.318755 I | op-k8sutil: ROOK_CSIADDONS_IMAGE="quay.io/csiaddons/k8s-sidecar:v0.2.1" (default) 2022-03-08T23:09:20.323 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:20.318759 I | op-k8sutil: ROOK_CSI_CEPHFS_POD_LABELS="" (default) 2022-03-08T23:09:20.324 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:20.318765 I | op-k8sutil: ROOK_CSI_RBD_POD_LABELS="" (default) 2022-03-08T23:09:20.324 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:20.318770 I | ceph-csi: detecting the ceph csi image version for image "quay.io/cephcsi/cephcsi:v3.5.1" 2022-03-08T23:09:20.324 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:20.318818 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-03-08T23:09:20.324 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:20.318828 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-08T23:09:20.763 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:20.761320 I | ceph-spec: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3... 2022-03-08T23:09:23.109 INFO:teuthology.orchestra.run.smithi077.stdout:Success 2022-03-08T23:09:23.131 INFO:teuthology.orchestra.run.smithi077.stderr:+ removed_pv= 2022-03-08T23:09:23.131 INFO:teuthology.orchestra.run.smithi077.stderr:+ '[' '' = '' ']' 2022-03-08T23:09:23.132 INFO:teuthology.orchestra.run.smithi077.stderr:++ kubectl get pv 2022-03-08T23:09:23.132 INFO:teuthology.orchestra.run.smithi077.stderr:++ grep Released 2022-03-08T23:09:23.132 INFO:teuthology.orchestra.run.smithi077.stderr:++ cut -f1 -d ' ' 2022-03-08T23:09:23.194 INFO:teuthology.orchestra.run.smithi077.stderr:+ removed_pv= 2022-03-08T23:09:23.194 INFO:teuthology.orchestra.run.smithi077.stderr:+ sleep 3s 2022-03-08T23:09:25.116 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:25.114811 I | ceph-spec: object "rook-ceph-osd-0" matched on delete, reconciling 2022-03-08T23:09:26.196 INFO:teuthology.orchestra.run.smithi077.stderr:+ '[' '' = '' ']' 2022-03-08T23:09:26.197 INFO:teuthology.orchestra.run.smithi077.stderr:++ kubectl get pv 2022-03-08T23:09:26.197 INFO:teuthology.orchestra.run.smithi077.stderr:++ grep Released 2022-03-08T23:09:26.197 INFO:teuthology.orchestra.run.smithi077.stderr:++ cut -f1 -d ' ' 2022-03-08T23:09:26.427 INFO:teuthology.orchestra.run.smithi077.stderr:+ removed_pv=smithi077-lv-3 2022-03-08T23:09:26.427 INFO:teuthology.orchestra.run.smithi077.stderr:+ sleep 3s 2022-03-08T23:09:27.320 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:27.290911 I | ceph-csi: Detected ceph CSI image version: "v3.5.1" 2022-03-08T23:09:27.320 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:27.290939 I | op-k8sutil: CSI_FORCE_CEPHFS_KERNEL_CLIENT="true" (configmap) 2022-03-08T23:09:27.320 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:27.290945 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default) 2022-03-08T23:09:27.321 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:27.290949 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default) 2022-03-08T23:09:27.321 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:27.290953 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default) 2022-03-08T23:09:27.321 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:27.290958 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default) 2022-03-08T23:09:27.321 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:27.290962 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default) 2022-03-08T23:09:27.322 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:27.290966 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default) 2022-03-08T23:09:27.322 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:27.290969 I | op-k8sutil: CSIADDONS_PORT="9070" (default) 2022-03-08T23:09:27.322 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:27.290972 I | op-k8sutil: CSIADDONS_PORT="9070" (default) 2022-03-08T23:09:27.322 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:27.290975 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default) 2022-03-08T23:09:27.322 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:27.290978 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default) 2022-03-08T23:09:27.323 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:27.290981 I | op-k8sutil: CSI_PLUGIN_PRIORITY_CLASSNAME="" (default) 2022-03-08T23:09:27.323 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:27.290984 I | op-k8sutil: CSI_PROVISIONER_PRIORITY_CLASSNAME="" (default) 2022-03-08T23:09:27.323 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:27.290987 I | op-k8sutil: CSI_ENABLE_OMAP_GENERATOR="false" (default) 2022-03-08T23:09:27.323 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:27.290991 I | op-k8sutil: CSI_ENABLE_RBD_SNAPSHOTTER="true" (configmap) 2022-03-08T23:09:27.324 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:27.290993 I | op-k8sutil: CSI_ENABLE_CEPHFS_SNAPSHOTTER="true" (configmap) 2022-03-08T23:09:27.324 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:27.290996 I | op-k8sutil: CSI_ENABLE_VOLUME_REPLICATION="false" (configmap) 2022-03-08T23:09:27.324 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:27.290999 I | op-k8sutil: CSI_ENABLE_CSIADDONS="false" (configmap) 2022-03-08T23:09:27.324 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:27.291004 I | op-k8sutil: CSI_CEPHFS_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-03-08T23:09:27.324 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:27.291008 I | op-k8sutil: CSI_RBD_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-03-08T23:09:27.325 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:27.291013 I | op-k8sutil: CSI_PLUGIN_ENABLE_SELINUX_HOST_MOUNT="false" (configmap) 2022-03-08T23:09:27.325 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:27.291018 I | ceph-csi: Kubernetes version is 1.23 2022-03-08T23:09:27.325 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:27.291024 I | op-k8sutil: ROOK_CSI_RESIZER_IMAGE="k8s.gcr.io/sig-storage/csi-resizer:v1.4.0" (default) 2022-03-08T23:09:27.325 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:27.291028 I | op-k8sutil: CSI_LOG_LEVEL="" (default) 2022-03-08T23:09:27.520 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:27.518366 I | op-k8sutil: CSI_PROVISIONER_REPLICAS="2" (configmap) 2022-03-08T23:09:27.529 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:27.527140 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-03-08T23:09:27.529 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:27.527157 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-08T23:09:27.529 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:27.527162 I | op-k8sutil: CSI_PLUGIN_TOLERATIONS="" (default) 2022-03-08T23:09:27.530 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:27.527165 I | op-k8sutil: CSI_PLUGIN_NODE_AFFINITY="" (default) 2022-03-08T23:09:27.530 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:27.527168 I | op-k8sutil: CSI_RBD_PLUGIN_TOLERATIONS="" (default) 2022-03-08T23:09:27.530 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:27.527171 I | op-k8sutil: CSI_RBD_PLUGIN_NODE_AFFINITY="" (default) 2022-03-08T23:09:27.530 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:27.527174 I | op-k8sutil: CSI_RBD_PLUGIN_RESOURCE="" (default) 2022-03-08T23:09:28.012 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:28.010329 I | op-k8sutil: CSI_RBD_PROVISIONER_TOLERATIONS="" (default) 2022-03-08T23:09:28.013 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:28.010365 I | op-k8sutil: CSI_RBD_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-08T23:09:28.013 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:28.010375 I | op-k8sutil: CSI_RBD_PROVISIONER_RESOURCE="" (default) 2022-03-08T23:09:28.146 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:28.144128 I | ceph-csi: successfully started CSI Ceph RBD driver 2022-03-08T23:09:28.364 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:28.361957 I | op-k8sutil: CSI_CEPHFS_PLUGIN_TOLERATIONS="" (default) 2022-03-08T23:09:28.364 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:28.361989 I | op-k8sutil: CSI_CEPHFS_PLUGIN_NODE_AFFINITY="" (default) 2022-03-08T23:09:28.364 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:28.361999 I | op-k8sutil: CSI_CEPHFS_PLUGIN_RESOURCE="" (default) 2022-03-08T23:09:28.491 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:28.488880 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_TOLERATIONS="" (default) 2022-03-08T23:09:28.491 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:28.488922 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-08T23:09:28.491 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:28.488937 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_RESOURCE="" (default) 2022-03-08T23:09:28.499 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:28.496866 I | ceph-spec: detected ceph image version: "17.0.0-11025 quincy" 2022-03-08T23:09:28.499 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:28.496893 I | ceph-cluster-controller: validating ceph version from provided image 2022-03-08T23:09:28.499 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:28.496908 W | ceph-cluster-controller: unsupported ceph version detected: "17.0.0-11025 quincy", pursuing 2022-03-08T23:09:29.071 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:29.069407 I | op-mon: parsing mon endpoints: a=10.110.141.123:6789,b=10.98.41.230:6789,c=10.109.47.142:6789 2022-03-08T23:09:29.076 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:29.070374 I | ceph-csi: successfully started CSI CephFS driver 2022-03-08T23:09:29.090 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:29.088012 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-08T23:09:29.091 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:29.088198 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-08T23:09:29.418 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:29.416796 I | op-k8sutil: CSI_RBD_FSGROUPPOLICY="ReadWriteOnceWithFSType" (configmap) 2022-03-08T23:09:29.422 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:29.420876 I | ceph-csi: CSIDriver object updated for driver "rook-ceph.rbd.csi.ceph.com" 2022-03-08T23:09:29.422 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:29.420889 I | op-k8sutil: CSI_CEPHFS_FSGROUPPOLICY="ReadWriteOnceWithFSType" (configmap) 2022-03-08T23:09:29.428 INFO:teuthology.orchestra.run.smithi077.stderr:+ '[' smithi077-lv-3 = '' ']' 2022-03-08T23:09:29.428 INFO:teuthology.orchestra.run.smithi077.stderr:++ kubectl get pv smithi077-lv-3 -o 'jsonpath={.spec.local.path}' 2022-03-08T23:09:29.448 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:29.446345 I | ceph-csi: CSIDriver object updated for driver "rook-ceph.cephfs.csi.ceph.com" 2022-03-08T23:09:29.483 INFO:teuthology.orchestra.run.smithi077.stderr:+ target_path=/dev/vg_nvme/lv_3 2022-03-08T23:09:29.484 INFO:teuthology.orchestra.run.smithi077.stderr:++ echo smithi077-lv-3 2022-03-08T23:09:29.484 INFO:teuthology.orchestra.run.smithi077.stderr:++ cut -f1 -d - 2022-03-08T23:09:29.485 INFO:teuthology.orchestra.run.smithi077.stderr:+ host=smithi077 2022-03-08T23:09:29.485 INFO:teuthology.orchestra.run.smithi077.stderr:+ toolbox ceph orch device zap smithi077 /dev/vg_nvme/lv_3 --force 2022-03-08T23:09:29.486 INFO:teuthology.orchestra.run.smithi077.stderr:+ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph orch device zap smithi077 /dev/vg_nvme/lv_3 --force 2022-03-08T23:09:29.562 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:29.560679 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-11025 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3" 2022-03-08T23:09:29.619 INFO:teuthology.orchestra.run.smithi077.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-08T23:09:29.794 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:29.792868 I | op-mon: start running mons 2022-03-08T23:09:29.799 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:29.797670 I | op-mon: parsing mon endpoints: a=10.110.141.123:6789,b=10.98.41.230:6789,c=10.109.47.142:6789 2022-03-08T23:09:30.211 INFO:teuthology.orchestra.run.smithi077.stdout:/dev/vg_nvme/lv_3 on smithi077 zapped 2022-03-08T23:09:30.232 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:30.230556 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.110.141.123:6789","10.98.41.230:6789","10.109.47.142:6789"]}] data:a=10.110.141.123:6789,b=10.98.41.230:6789,c=10.109.47.142:6789 mapping:{"node":{"a":{"Name":"smithi135","Hostname":"smithi135","Address":"172.21.15.135"},"b":{"Name":"smithi106","Hostname":"smithi106","Address":"172.21.15.106"},"c":{"Name":"smithi077","Hostname":"smithi077","Address":"172.21.15.77"}}} maxMonId:2] 2022-03-08T23:09:30.233 INFO:teuthology.orchestra.run.smithi077.stderr:+ zap_completion=0 2022-03-08T23:09:30.234 INFO:teuthology.orchestra.run.smithi077.stderr:+ '[' 0 = 0 ']' 2022-03-08T23:09:30.234 INFO:teuthology.orchestra.run.smithi077.stderr:++ kubectl get job -n rook-ceph rook-ceph-device-zap -o 'jsonpath={.status.succeeded.path}' 2022-03-08T23:09:30.392 INFO:teuthology.orchestra.run.smithi077.stderr:+ zap_completion= 2022-03-08T23:09:30.392 INFO:teuthology.orchestra.run.smithi077.stderr:+ sleep 3s 2022-03-08T23:09:30.550 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:30.548641 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-08T23:09:30.553 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:30.548981 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-08T23:09:32.150 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:32.148794 I | op-mon: targeting the mon count 3 2022-03-08T23:09:32.158 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:32.156356 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-08T23:09:32.491 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:32.489430 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-08T23:09:32.492 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:32.489463 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-03-08T23:09:32.826 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:32.823951 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-03-08T23:09:32.826 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:32.823989 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-08T23:09:33.366 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:33.364310 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-08T23:09:33.366 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:33.364339 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-03-08T23:09:33.393 INFO:teuthology.orchestra.run.smithi077.stderr:+ '[' '' = 0 ']' 2022-03-08T23:09:33.393 INFO:teuthology.orchestra.run.smithi077.stderr:+ kubectl patch pv smithi077-lv-3 -p '{"spec":{"claimRef": null}}' 2022-03-08T23:09:33.557 INFO:teuthology.orchestra.run.smithi077.stdout:persistentvolume/smithi077-lv-3 patched 2022-03-08T23:09:33.561 INFO:teuthology.orchestra.run.smithi077.stderr:+ toolbox ceph orch apply osd --all-available-devices 2022-03-08T23:09:33.562 INFO:teuthology.orchestra.run.smithi077.stderr:+ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph orch apply osd --all-available-devices 2022-03-08T23:09:33.782 INFO:teuthology.orchestra.run.smithi077.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-08T23:09:33.786 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:33.782326 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-03-08T23:09:33.786 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:33.782354 I | op-config: deleting "log file" option from the mon configuration database 2022-03-08T23:09:34.160 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:34.158178 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-03-08T23:09:34.160 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:34.158205 I | op-mon: checking for basic quorum with existing mons 2022-03-08T23:09:34.305 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:34.303968 I | op-mon: mon "a" endpoint is [v2:10.110.141.123:3300,v1:10.110.141.123:6789] 2022-03-08T23:09:35.023 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:35.021132 I | op-mon: mon "b" endpoint is [v2:10.98.41.230:3300,v1:10.98.41.230:6789] 2022-03-08T23:09:35.027 INFO:teuthology.orchestra.run.smithi077.stdout:['Success'] 2022-03-08T23:09:35.033 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:35.031634 I | ceph-cluster-controller: CR has changed for "rook-ceph". diff=  v1.ClusterSpec{ 2022-03-08T23:09:35.033 INFO:tasks.rook.operator.smithi077.stdout:   CephVersion: {Image: "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", AllowUnsupported: true}, 2022-03-08T23:09:35.034 INFO:tasks.rook.operator.smithi077.stdout:   Storage: v1.StorageScopeSpec{ 2022-03-08T23:09:35.034 INFO:tasks.rook.operator.smithi077.stdout:   ... // 3 identical fields 2022-03-08T23:09:35.036 INFO:tasks.rook.operator.smithi077.stdout:   Config: nil, 2022-03-08T23:09:35.036 INFO:tasks.rook.operator.smithi077.stdout:   Selection: {}, 2022-03-08T23:09:35.036 INFO:tasks.rook.operator.smithi077.stdout:   StorageClassDeviceSets: []v1.StorageClassDeviceSet{ 2022-03-08T23:09:35.037 INFO:tasks.rook.operator.smithi077.stdout:   ... // 9 identical elements 2022-03-08T23:09:35.037 INFO:tasks.rook.operator.smithi077.stdout:   {Name: "smithi135-lv-3", Count: 1, VolumeClaimTemplates: {{ObjectMeta: {Name: "data"}, Spec: {AccessModes: {"ReadWriteOnce"}, Resources: {Requests: {s"storage": {i: {...}, s: "1", Format: "DecimalSI"}}}, VolumeName: "smithi135-lv-3", StorageClassName: &"scratch", ...}}}}, 2022-03-08T23:09:35.037 INFO:tasks.rook.operator.smithi077.stdout:   {Name: "smithi135-lv-4", Count: 1, VolumeClaimTemplates: {{ObjectMeta: {Name: "data"}, Spec: {AccessModes: {"ReadWriteOnce"}, Resources: {Requests: {s"storage": {i: {...}, s: "1", Format: "DecimalSI"}}}, VolumeName: "smithi135-lv-4", StorageClassName: &"scratch", ...}}}}, 2022-03-08T23:09:35.037 INFO:tasks.rook.operator.smithi077.stdout:+  { 2022-03-08T23:09:35.037 INFO:tasks.rook.operator.smithi077.stdout:+  Name: "smithi077-lv-3", 2022-03-08T23:09:35.038 INFO:tasks.rook.operator.smithi077.stdout:+  Count: 1, 2022-03-08T23:09:35.038 INFO:tasks.rook.operator.smithi077.stdout:+  VolumeClaimTemplates: []v1.PersistentVolumeClaim{{ObjectMeta: v1.ObjectMeta{...}, Spec: v1.PersistentVolumeClaimSpec{...}}}, 2022-03-08T23:09:35.038 INFO:tasks.rook.operator.smithi077.stdout:+  }, 2022-03-08T23:09:35.038 INFO:tasks.rook.operator.smithi077.stdout:   }, 2022-03-08T23:09:35.038 INFO:tasks.rook.operator.smithi077.stdout:   }, 2022-03-08T23:09:35.039 INFO:tasks.rook.operator.smithi077.stdout:   Annotations: nil, 2022-03-08T23:09:35.039 INFO:tasks.rook.operator.smithi077.stdout:   Labels: nil, 2022-03-08T23:09:35.039 INFO:tasks.rook.operator.smithi077.stdout:   ... // 20 identical fields 2022-03-08T23:09:35.039 INFO:tasks.rook.operator.smithi077.stdout:  } 2022-03-08T23:09:35.040 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:35.031724 I | operator: reloading operator's CRDs manager, cancelling all orchestrations! 2022-03-08T23:09:35.040 INFO:tasks.rook.operator.smithi077.stdout:I0308 23:09:35.031964 1 manager.go:148] objectbucket.io/provisioner-manager "msg"="stopping provisioner" "name"="rook-ceph.ceph.rook.io/bucket" "reason"={} 2022-03-08T23:09:35.041 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:35.036428 I | op-k8sutil: ROOK_CURRENT_NAMESPACE_ONLY="false" (env var) 2022-03-08T23:09:35.041 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:35.036454 I | operator: watching all namespaces for Ceph CRs 2022-03-08T23:09:35.041 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:35.036493 I | operator: setting up schemes 2022-03-08T23:09:35.041 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:35.038642 I | operator: setting up the controller-runtime manager 2022-03-08T23:09:35.051 INFO:teuthology.orchestra.run.smithi077.stderr:+ kubectl delete job rook-ceph-device-zap -n rook-ceph 2022-03-08T23:09:35.200 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:35.198235 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". failed to reconcile cluster "rook-ceph": failed to configure local ceph cluster: failed to create cluster: failed to start ceph monitors: failed to init mon services: failed to create mon service: failed to create service for mon c: failed to create service rook-ceph-mon-c. context canceled 2022-03-08T23:09:35.200 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:35.198376 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-03-08T23:09:35.217 INFO:teuthology.orchestra.run.smithi077.stdout:job.batch "rook-ceph-device-zap" deleted 2022-03-08T23:09:35.244 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:35.242102 I | operator: looking for admission webhook secret "rook-ceph-admission-controller" 2022-03-08T23:09:35.244 INFO:teuthology.orchestra.run.smithi077.stderr:+ num_osd=0 2022-03-08T23:09:35.245 INFO:teuthology.orchestra.run.smithi077.stderr:+ '[' 0 '!=' 12 ']' 2022-03-08T23:09:35.245 INFO:teuthology.orchestra.run.smithi077.stdout:waiting for osd to come back up 2022-03-08T23:09:35.246 INFO:teuthology.orchestra.run.smithi077.stderr:+ echo 'waiting for osd to come back up' 2022-03-08T23:09:35.246 INFO:teuthology.orchestra.run.smithi077.stderr:++ cut -f3 -d ' ' 2022-03-08T23:09:35.246 INFO:teuthology.orchestra.run.smithi077.stderr:++ toolbox ceph osd stat 2022-03-08T23:09:35.247 INFO:teuthology.orchestra.run.smithi077.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-03-08T23:09:35.252 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:35.242719 I | op-mon: parsing mon endpoints: a=10.110.141.123:6789,b=10.98.41.230:6789,c=10.109.47.142:6789 2022-03-08T23:09:35.253 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:35.244546 I | operator: admission webhook secret "rook-ceph-admission-controller" not found. proceeding without the admission controller 2022-03-08T23:09:35.253 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:35.244585 I | ceph-cluster-controller: successfully started 2022-03-08T23:09:35.253 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:35.244644 I | ceph-cluster-controller: enabling hotplug orchestration 2022-03-08T23:09:35.253 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:35.244666 I | ceph-crashcollector-controller: successfully started 2022-03-08T23:09:35.254 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:35.244687 I | ceph-block-pool-controller: successfully started 2022-03-08T23:09:35.254 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:35.244710 I | ceph-object-store-user-controller: successfully started 2022-03-08T23:09:35.254 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:35.244731 I | ceph-object-realm-controller: successfully started 2022-03-08T23:09:35.254 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:35.244748 I | ceph-object-zonegroup-controller: successfully started 2022-03-08T23:09:35.254 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:35.244760 I | ceph-object-zone-controller: successfully started 2022-03-08T23:09:35.255 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:35.244899 I | ceph-object-controller: successfully started 2022-03-08T23:09:35.255 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:35.244936 I | ceph-file-controller: successfully started 2022-03-08T23:09:35.255 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:35.244968 I | ceph-nfs-controller: successfully started 2022-03-08T23:09:35.255 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:35.244997 I | ceph-rbd-mirror-controller: successfully started 2022-03-08T23:09:35.256 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:35.245024 I | ceph-client-controller: successfully started 2022-03-08T23:09:35.256 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:35.245046 I | ceph-filesystem-mirror-controller: successfully started 2022-03-08T23:09:35.256 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:35.245072 I | operator: rook-ceph-operator-config-controller successfully started 2022-03-08T23:09:35.256 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:35.245091 I | ceph-csi: rook-ceph-operator-csi-controller successfully started 2022-03-08T23:09:35.256 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:35.245107 I | op-bucket-prov: rook-ceph-operator-bucket-controller successfully started 2022-03-08T23:09:35.257 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:35.245123 I | ceph-bucket-topic: successfully started 2022-03-08T23:09:35.257 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:35.245139 I | ceph-bucket-notification: successfully started 2022-03-08T23:09:35.257 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:35.245150 I | ceph-bucket-notification: successfully started 2022-03-08T23:09:35.257 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:35.245164 I | ceph-fs-subvolumegroup-controller: successfully started 2022-03-08T23:09:35.258 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:35.246171 I | operator: starting the controller-runtime manager 2022-03-08T23:09:35.353 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:35.351145 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-03-08T23:09:35.357 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:35.351671 I | op-k8sutil: ROOK_CEPH_COMMANDS_TIMEOUT_SECONDS="15" (configmap) 2022-03-08T23:09:35.358 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:35.351754 I | op-k8sutil: ROOK_LOG_LEVEL="INFO" (configmap) 2022-03-08T23:09:35.358 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:35.351779 I | op-k8sutil: ROOK_ENABLE_DISCOVERY_DAEMON="false" (configmap) 2022-03-08T23:09:35.365 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:35.363606 I | operator: rook-ceph-operator-config-controller done reconciling 2022-03-08T23:09:35.368 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:35.365487 I | ceph-spec: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3... 2022-03-08T23:09:35.376 INFO:teuthology.orchestra.run.smithi077.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-08T23:09:35.564 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:35.562106 I | op-mon: parsing mon endpoints: a=10.110.141.123:6789,b=10.98.41.230:6789,c=10.109.47.142:6789 2022-03-08T23:09:35.564 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:35.562199 I | op-k8sutil: ROOK_OBC_WATCH_OPERATOR_NAMESPACE="true" (configmap) 2022-03-08T23:09:35.564 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:35.562213 I | op-bucket-prov: ceph bucket provisioner launched watching for provisioner "rook-ceph.ceph.rook.io/bucket" 2022-03-08T23:09:35.569 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:35.564626 I | op-bucket-prov: successfully reconciled bucket provisioner 2022-03-08T23:09:35.569 INFO:tasks.rook.operator.smithi077.stdout:I0308 23:09:35.564725 1 manager.go:135] objectbucket.io/provisioner-manager "msg"="starting provisioner" "name"="rook-ceph.ceph.rook.io/bucket" 2022-03-08T23:09:35.763 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:35.761534 I | op-mon: parsing mon endpoints: a=10.110.141.123:6789,b=10.98.41.230:6789,c=10.109.47.142:6789 2022-03-08T23:09:35.784 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:35.782612 I | ceph-spec: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3... 2022-03-08T23:09:35.874 INFO:teuthology.orchestra.run.smithi077.stderr:+ num_osd=11 2022-03-08T23:09:35.875 INFO:teuthology.orchestra.run.smithi077.stderr:+ sleep 30s 2022-03-08T23:09:36.436 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:36.433994 I | ceph-csi: successfully created csi config map "rook-ceph-csi-config" 2022-03-08T23:09:36.523 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:36.521227 I | op-k8sutil: Removing previous job rook-ceph-detect-version to start a new one 2022-03-08T23:09:36.579 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:36.577272 I | op-k8sutil: batch job rook-ceph-detect-version still exists 2022-03-08T23:09:36.790 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:36.786768 I | op-k8sutil: ROOK_CSI_ENABLE_RBD="true" (configmap) 2022-03-08T23:09:36.791 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:36.786799 I | op-k8sutil: ROOK_CSI_ENABLE_CEPHFS="true" (configmap) 2022-03-08T23:09:36.791 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:36.786808 I | op-k8sutil: ROOK_CSI_ALLOW_UNSUPPORTED_VERSION="false" (configmap) 2022-03-08T23:09:36.791 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:36.786816 I | op-k8sutil: ROOK_CSI_ENABLE_GRPC_METRICS="false" (configmap) 2022-03-08T23:09:36.791 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:36.786823 I | op-k8sutil: CSI_ENABLE_HOST_NETWORK="true" (default) 2022-03-08T23:09:36.791 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:36.786832 I | op-k8sutil: ROOK_CSI_CEPH_IMAGE="quay.io/cephcsi/cephcsi:v3.5.1" (default) 2022-03-08T23:09:36.792 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:36.786841 I | op-k8sutil: ROOK_CSI_REGISTRAR_IMAGE="k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.5.0" (default) 2022-03-08T23:09:36.792 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:36.786857 I | op-k8sutil: ROOK_CSI_PROVISIONER_IMAGE="k8s.gcr.io/sig-storage/csi-provisioner:v3.1.0" (default) 2022-03-08T23:09:36.792 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:36.786870 I | op-k8sutil: ROOK_CSI_ATTACHER_IMAGE="k8s.gcr.io/sig-storage/csi-attacher:v3.4.0" (default) 2022-03-08T23:09:36.792 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:36.786878 I | op-k8sutil: ROOK_CSI_SNAPSHOTTER_IMAGE="k8s.gcr.io/sig-storage/csi-snapshotter:v5.0.1" (default) 2022-03-08T23:09:36.792 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:36.786886 I | op-k8sutil: ROOK_CSI_KUBELET_DIR_PATH="/var/lib/kubelet" (default) 2022-03-08T23:09:36.793 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:36.786896 I | op-k8sutil: CSI_VOLUME_REPLICATION_IMAGE="quay.io/csiaddons/volumereplication-operator:v0.3.0" (default) 2022-03-08T23:09:36.793 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:36.786905 I | op-k8sutil: ROOK_CSIADDONS_IMAGE="quay.io/csiaddons/k8s-sidecar:v0.2.1" (default) 2022-03-08T23:09:36.793 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:36.786912 I | op-k8sutil: ROOK_CSI_CEPHFS_POD_LABELS="" (default) 2022-03-08T23:09:36.793 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:36.786921 I | op-k8sutil: ROOK_CSI_RBD_POD_LABELS="" (default) 2022-03-08T23:09:36.794 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:36.786936 I | ceph-csi: detecting the ceph csi image version for image "quay.io/cephcsi/cephcsi:v3.5.1" 2022-03-08T23:09:36.794 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:36.787026 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-03-08T23:09:36.794 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:36.787046 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-08T23:09:39.585 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:39.583018 I | op-k8sutil: batch job rook-ceph-detect-version still exists 2022-03-08T23:09:42.563 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:42.559633 I | ceph-csi: Detected ceph CSI image version: "v3.5.1" 2022-03-08T23:09:42.563 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:42.559675 I | op-k8sutil: CSI_FORCE_CEPHFS_KERNEL_CLIENT="true" (configmap) 2022-03-08T23:09:42.563 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:42.559688 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default) 2022-03-08T23:09:42.564 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:42.559695 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default) 2022-03-08T23:09:42.564 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:42.559704 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default) 2022-03-08T23:09:42.564 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:42.559711 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default) 2022-03-08T23:09:42.564 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:42.559718 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default) 2022-03-08T23:09:42.565 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:42.559729 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default) 2022-03-08T23:09:42.565 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:42.559737 I | op-k8sutil: CSIADDONS_PORT="9070" (default) 2022-03-08T23:09:42.565 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:42.559748 I | op-k8sutil: CSIADDONS_PORT="9070" (default) 2022-03-08T23:09:42.565 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:42.559755 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default) 2022-03-08T23:09:42.565 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:42.559782 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default) 2022-03-08T23:09:42.566 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:42.559789 I | op-k8sutil: CSI_PLUGIN_PRIORITY_CLASSNAME="" (default) 2022-03-08T23:09:42.566 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:42.559796 I | op-k8sutil: CSI_PROVISIONER_PRIORITY_CLASSNAME="" (default) 2022-03-08T23:09:42.566 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:42.559803 I | op-k8sutil: CSI_ENABLE_OMAP_GENERATOR="false" (default) 2022-03-08T23:09:42.566 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:42.559809 I | op-k8sutil: CSI_ENABLE_RBD_SNAPSHOTTER="true" (configmap) 2022-03-08T23:09:42.567 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:42.559816 I | op-k8sutil: CSI_ENABLE_CEPHFS_SNAPSHOTTER="true" (configmap) 2022-03-08T23:09:42.567 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:42.559825 I | op-k8sutil: CSI_ENABLE_VOLUME_REPLICATION="false" (configmap) 2022-03-08T23:09:42.567 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:42.559835 I | op-k8sutil: CSI_ENABLE_CSIADDONS="false" (configmap) 2022-03-08T23:09:42.567 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:42.559846 I | op-k8sutil: CSI_CEPHFS_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-03-08T23:09:42.567 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:42.559856 I | op-k8sutil: CSI_RBD_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-03-08T23:09:42.568 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:42.559866 I | op-k8sutil: CSI_PLUGIN_ENABLE_SELINUX_HOST_MOUNT="false" (configmap) 2022-03-08T23:09:42.568 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:42.559874 I | ceph-csi: Kubernetes version is 1.23 2022-03-08T23:09:42.568 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:42.559882 I | op-k8sutil: ROOK_CSI_RESIZER_IMAGE="k8s.gcr.io/sig-storage/csi-resizer:v1.4.0" (default) 2022-03-08T23:09:42.568 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:42.559889 I | op-k8sutil: CSI_LOG_LEVEL="" (default) 2022-03-08T23:09:42.579 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:42.568579 I | op-k8sutil: CSI_PROVISIONER_REPLICAS="2" (configmap) 2022-03-08T23:09:42.579 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:42.576667 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-03-08T23:09:42.580 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:42.576687 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-08T23:09:42.580 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:42.576691 I | op-k8sutil: CSI_PLUGIN_TOLERATIONS="" (default) 2022-03-08T23:09:42.580 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:42.576694 I | op-k8sutil: CSI_PLUGIN_NODE_AFFINITY="" (default) 2022-03-08T23:09:42.580 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:42.576697 I | op-k8sutil: CSI_RBD_PLUGIN_TOLERATIONS="" (default) 2022-03-08T23:09:42.580 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:42.576699 I | op-k8sutil: CSI_RBD_PLUGIN_NODE_AFFINITY="" (default) 2022-03-08T23:09:42.581 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:42.576702 I | op-k8sutil: CSI_RBD_PLUGIN_RESOURCE="" (default) 2022-03-08T23:09:42.663 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:42.661674 I | op-k8sutil: batch job rook-ceph-detect-version deleted 2022-03-08T23:09:42.754 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:42.752982 I | op-k8sutil: CSI_RBD_PROVISIONER_TOLERATIONS="" (default) 2022-03-08T23:09:42.755 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:42.753014 I | op-k8sutil: CSI_RBD_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-08T23:09:42.755 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:42.753029 I | op-k8sutil: CSI_RBD_PROVISIONER_RESOURCE="" (default) 2022-03-08T23:09:42.836 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:42.834221 I | ceph-csi: successfully started CSI Ceph RBD driver 2022-03-08T23:09:43.015 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:43.013846 I | op-k8sutil: CSI_CEPHFS_PLUGIN_TOLERATIONS="" (default) 2022-03-08T23:09:43.016 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:43.013865 I | op-k8sutil: CSI_CEPHFS_PLUGIN_NODE_AFFINITY="" (default) 2022-03-08T23:09:43.016 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:43.013870 I | op-k8sutil: CSI_CEPHFS_PLUGIN_RESOURCE="" (default) 2022-03-08T23:09:43.099 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:43.097570 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_TOLERATIONS="" (default) 2022-03-08T23:09:43.099 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:43.097599 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-08T23:09:43.100 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:43.097608 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_RESOURCE="" (default) 2022-03-08T23:09:43.178 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:43.176841 I | ceph-csi: successfully started CSI CephFS driver 2022-03-08T23:09:43.384 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:43.383038 I | op-k8sutil: CSI_RBD_FSGROUPPOLICY="ReadWriteOnceWithFSType" (configmap) 2022-03-08T23:09:43.429 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:43.428055 I | ceph-csi: CSIDriver object updated for driver "rook-ceph.rbd.csi.ceph.com" 2022-03-08T23:09:43.430 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:43.428086 I | op-k8sutil: CSI_CEPHFS_FSGROUPPOLICY="ReadWriteOnceWithFSType" (configmap) 2022-03-08T23:09:43.435 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:43.433481 I | ceph-csi: CSIDriver object updated for driver "rook-ceph.cephfs.csi.ceph.com" 2022-03-08T23:09:48.158 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:48.156317 I | ceph-spec: detected ceph image version: "17.0.0-11025 quincy" 2022-03-08T23:09:48.158 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:48.156345 I | ceph-cluster-controller: validating ceph version from provided image 2022-03-08T23:09:48.159 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:48.156357 W | ceph-cluster-controller: unsupported ceph version detected: "17.0.0-11025 quincy", pursuing 2022-03-08T23:09:48.214 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:48.212772 I | ceph-spec: detected ceph image version: "17.0.0-11025 quincy" 2022-03-08T23:09:48.214 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:48.212801 I | ceph-cluster-controller: validating ceph version from provided image 2022-03-08T23:09:48.215 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:48.212815 W | ceph-cluster-controller: unsupported ceph version detected: "17.0.0-11025 quincy", pursuing 2022-03-08T23:09:48.219 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:48.214663 I | op-mon: parsing mon endpoints: a=10.110.141.123:6789,b=10.98.41.230:6789,c=10.109.47.142:6789 2022-03-08T23:09:48.244 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:48.243208 I | op-mon: parsing mon endpoints: a=10.110.141.123:6789,b=10.98.41.230:6789,c=10.109.47.142:6789 2022-03-08T23:09:48.270 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:48.268842 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-08T23:09:48.270 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:48.269201 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-08T23:09:48.279 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:48.277765 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-08T23:09:48.282 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:48.278028 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-08T23:09:48.700 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:48.698154 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-11025 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3" 2022-03-08T23:09:48.700 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:48.698195 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-11025 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3" 2022-03-08T23:09:48.807 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:48.806162 E | ceph-spec: failed to update cluster condition to {Type:Progressing Status:True Reason:ClusterProgressing Message:Configuring the Ceph cluster LastHeartbeatTime:2022-03-08 23:09:48.718396533 +0000 UTC m=+514.535122978 LastTransitionTime:2022-03-08 23:09:48.718396409 +0000 UTC m=+514.535122866}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-08T23:09:48.834 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:48.832817 I | op-mon: start running mons 2022-03-08T23:09:48.859 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:48.857699 E | ceph-spec: failed to update cluster condition to {Type:Progressing Status:True Reason:ClusterProgressing Message:Configuring Ceph Mons LastHeartbeatTime:2022-03-08 23:09:48.827700433 +0000 UTC m=+514.644426884 LastTransitionTime:2022-03-08 23:09:48.827700267 +0000 UTC m=+514.644426738}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-08T23:09:48.894 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:48.893696 I | op-mon: parsing mon endpoints: a=10.110.141.123:6789,b=10.98.41.230:6789,c=10.109.47.142:6789 2022-03-08T23:09:49.886 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:49.884776 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.109.47.142:6789","10.110.141.123:6789","10.98.41.230:6789"]}] data:c=10.109.47.142:6789,a=10.110.141.123:6789,b=10.98.41.230:6789 mapping:{"node":{"a":{"Name":"smithi135","Hostname":"smithi135","Address":"172.21.15.135"},"b":{"Name":"smithi106","Hostname":"smithi106","Address":"172.21.15.106"},"c":{"Name":"smithi077","Hostname":"smithi077","Address":"172.21.15.77"}}} maxMonId:2] 2022-03-08T23:09:50.482 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:50.481052 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-08T23:09:50.486 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:50.481333 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-08T23:09:52.085 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:52.083635 I | op-mon: targeting the mon count 3 2022-03-08T23:09:52.093 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:52.092357 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-08T23:09:52.457 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:52.456325 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-08T23:09:52.458 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:52.456355 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-03-08T23:09:52.822 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:52.820954 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-03-08T23:09:52.823 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:52.820990 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-08T23:09:53.156 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:53.154704 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-08T23:09:53.156 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:53.154741 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-03-08T23:09:53.520 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:53.518657 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-03-08T23:09:53.521 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:53.518699 I | op-config: deleting "log file" option from the mon configuration database 2022-03-08T23:09:53.960 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:53.958553 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-03-08T23:09:53.960 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:53.958585 I | op-mon: checking for basic quorum with existing mons 2022-03-08T23:09:54.103 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:54.102084 I | op-mon: mon "a" endpoint is [v2:10.110.141.123:3300,v1:10.110.141.123:6789] 2022-03-08T23:09:54.203 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:54.202266 I | op-mon: mon "b" endpoint is [v2:10.98.41.230:3300,v1:10.98.41.230:6789] 2022-03-08T23:09:54.493 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:54.491790 I | op-mon: mon "c" endpoint is [v2:10.109.47.142:3300,v1:10.109.47.142:6789] 2022-03-08T23:09:55.137 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:55.136496 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.98.41.230:6789","10.109.47.142:6789","10.110.141.123:6789"]}] data:a=10.110.141.123:6789,b=10.98.41.230:6789,c=10.109.47.142:6789 mapping:{"node":{"a":{"Name":"smithi135","Hostname":"smithi135","Address":"172.21.15.135"},"b":{"Name":"smithi106","Hostname":"smithi106","Address":"172.21.15.106"},"c":{"Name":"smithi077","Hostname":"smithi077","Address":"172.21.15.77"}}} maxMonId:2] 2022-03-08T23:09:55.683 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:55.681685 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-08T23:09:55.687 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:55.681995 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-08T23:09:56.105 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:56.103599 I | op-mon: deployment for mon rook-ceph-mon-a already exists. updating if needed 2022-03-08T23:09:56.120 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:56.119149 I | op-k8sutil: deployment "rook-ceph-mon-a" did not change, nothing to update 2022-03-08T23:09:56.120 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:56.119188 I | op-mon: waiting for mon quorum with [a b c] 2022-03-08T23:09:56.744 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:56.742649 I | op-mon: mons running: [a b c] 2022-03-08T23:09:57.160 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:57.159093 I | op-mon: Monitors in quorum: [a b c] 2022-03-08T23:09:57.166 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:57.165742 I | op-mon: deployment for mon rook-ceph-mon-b already exists. updating if needed 2022-03-08T23:09:57.175 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:57.174784 I | op-k8sutil: deployment "rook-ceph-mon-b" did not change, nothing to update 2022-03-08T23:09:57.176 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:57.174807 I | op-mon: waiting for mon quorum with [a b c] 2022-03-08T23:09:57.289 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:57.288053 I | op-mon: mons running: [a b c] 2022-03-08T23:09:57.687 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:57.685892 I | op-mon: Monitors in quorum: [a b c] 2022-03-08T23:09:57.701 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:57.700475 I | op-mon: deployment for mon rook-ceph-mon-c already exists. updating if needed 2022-03-08T23:09:57.711 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:57.710114 I | op-k8sutil: deployment "rook-ceph-mon-c" did not change, nothing to update 2022-03-08T23:09:57.711 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:57.710135 I | op-mon: waiting for mon quorum with [a b c] 2022-03-08T23:09:57.892 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:57.891646 I | op-mon: mons running: [a b c] 2022-03-08T23:09:58.326 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:58.325089 I | op-mon: Monitors in quorum: [a b c] 2022-03-08T23:09:58.327 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:58.325119 I | op-mon: mons created: 3 2022-03-08T23:09:58.801 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:58.799747 I | op-mon: waiting for mon quorum with [a b c] 2022-03-08T23:09:58.837 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:58.836511 I | op-mon: mons running: [a b c] 2022-03-08T23:09:59.229 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:59.227934 I | op-mon: Monitors in quorum: [a b c] 2022-03-08T23:09:59.230 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:59.227994 I | cephclient: getting or creating ceph auth key "client.csi-rbd-provisioner" 2022-03-08T23:09:59.230 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:59.228062 I | op-mon: start running mons 2022-03-08T23:09:59.236 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:59.235800 I | op-mon: parsing mon endpoints: a=10.110.141.123:6789,b=10.98.41.230:6789,c=10.109.47.142:6789 2022-03-08T23:09:59.645 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:59.644532 I | cephclient: getting or creating ceph auth key "client.csi-rbd-node" 2022-03-08T23:09:59.884 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:09:59.883216 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.110.141.123:6789","10.98.41.230:6789","10.109.47.142:6789"]}] data:c=10.109.47.142:6789,a=10.110.141.123:6789,b=10.98.41.230:6789 mapping:{"node":{"a":{"Name":"smithi135","Hostname":"smithi135","Address":"172.21.15.135"},"b":{"Name":"smithi106","Hostname":"smithi106","Address":"172.21.15.106"},"c":{"Name":"smithi077","Hostname":"smithi077","Address":"172.21.15.77"}}} maxMonId:2] 2022-03-08T23:10:00.057 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:00.055874 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-provisioner" 2022-03-08T23:10:00.484 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:00.483068 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-08T23:10:00.489 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:00.483382 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-08T23:10:00.511 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:00.510397 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-node" 2022-03-08T23:10:03.085 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:03.083904 I | op-mon: targeting the mon count 3 2022-03-08T23:10:03.105 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:03.100121 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-08T23:10:03.459 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:03.458632 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-08T23:10:03.460 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:03.458665 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-03-08T23:10:03.683 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:03.682608 I | ceph-csi: created kubernetes csi secrets for cluster "rook-ceph" 2022-03-08T23:10:03.684 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:03.682640 I | cephclient: getting or creating ceph auth key "client.crash" 2022-03-08T23:10:03.894 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:03.892980 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-03-08T23:10:03.895 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:03.893010 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-08T23:10:04.144 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:04.143460 I | ceph-crashcollector-controller: created kubernetes crash collector secret for cluster "rook-ceph" 2022-03-08T23:10:04.346 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:04.344976 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-08T23:10:04.347 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:04.345006 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-03-08T23:10:04.606 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:04.604784 I | cephclient: successfully enabled msgr2 protocol 2022-03-08T23:10:04.606 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:04.604835 I | cephclient: create rbd-mirror bootstrap peer token "client.rbd-mirror-peer" 2022-03-08T23:10:04.606 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:04.604848 I | cephclient: getting or creating ceph auth key "client.rbd-mirror-peer" 2022-03-08T23:10:04.686 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:04.685238 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-03-08T23:10:04.687 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:04.685268 I | op-config: deleting "log file" option from the mon configuration database 2022-03-08T23:10:05.018 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:05.016785 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-03-08T23:10:05.018 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:05.016813 I | op-mon: checking for basic quorum with existing mons 2022-03-08T23:10:05.021 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:05.020193 I | cephclient: successfully created rbd-mirror bootstrap peer token for cluster "rook-ceph" 2022-03-08T23:10:05.117 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:05.116190 I | op-mgr: start running mgr 2022-03-08T23:10:05.117 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:05.116232 I | cephclient: getting or creating ceph auth key "mgr.a" 2022-03-08T23:10:05.283 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:05.281976 I | op-mon: mon "a" endpoint is [v2:10.110.141.123:3300,v1:10.110.141.123:6789] 2022-03-08T23:10:05.875 INFO:teuthology.orchestra.run.smithi077.stderr:+ '[' 11 '!=' 12 ']' 2022-03-08T23:10:05.875 INFO:teuthology.orchestra.run.smithi077.stderr:+ echo 'waiting for osd to come back up' 2022-03-08T23:10:05.876 INFO:teuthology.orchestra.run.smithi077.stdout:waiting for osd to come back up 2022-03-08T23:10:05.877 INFO:teuthology.orchestra.run.smithi077.stderr:++ toolbox ceph osd stat 2022-03-08T23:10:05.877 INFO:teuthology.orchestra.run.smithi077.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-03-08T23:10:05.877 INFO:teuthology.orchestra.run.smithi077.stderr:++ cut -f3 -d ' ' 2022-03-08T23:10:05.944 INFO:teuthology.orchestra.run.smithi077.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-08T23:10:06.486 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:06.485456 I | op-mon: mon "b" endpoint is [v2:10.98.41.230:3300,v1:10.98.41.230:6789] 2022-03-08T23:10:06.517 INFO:teuthology.orchestra.run.smithi077.stderr:+ num_osd=11 2022-03-08T23:10:06.518 INFO:teuthology.orchestra.run.smithi077.stderr:+ sleep 30s 2022-03-08T23:10:06.714 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:06.713524 I | op-mgr: deployment for mgr rook-ceph-mgr-a already exists. updating if needed 2022-03-08T23:10:06.725 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:06.725090 I | op-k8sutil: deployment "rook-ceph-mgr-a" did not change, nothing to update 2022-03-08T23:10:06.729 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:06.728243 I | op-mgr: setting services to point to mgr "a" 2022-03-08T23:10:08.086 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:08.085685 I | op-mon: mon "c" endpoint is [v2:10.109.47.142:3300,v1:10.109.47.142:6789] 2022-03-08T23:10:08.683 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:08.682738 I | op-mgr: no need to update service "rook-ceph-mgr" 2022-03-08T23:10:08.684 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:08.682820 I | op-mgr: successful modules: balancer 2022-03-08T23:10:08.698 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:08.697647 I | op-osd: start running osds in namespace "rook-ceph" 2022-03-08T23:10:08.698 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:08.697674 I | op-osd: wait timeout for healthy OSDs during upgrade or restart is "10m0s" 2022-03-08T23:10:08.754 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:08.750254 I | op-osd: start provisioning the OSDs on PVCs, if needed 2022-03-08T23:10:09.087 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:09.084286 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi077-lv-1" 2022-03-08T23:10:09.088 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:09.084314 I | op-osd: OSD PVC "smithi077-lv-1-data-0hb6kc" already exists 2022-03-08T23:10:09.088 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:09.084326 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi077-lv-2" 2022-03-08T23:10:09.088 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:09.084336 I | op-osd: OSD PVC "smithi077-lv-2-data-0hqxxt" already exists 2022-03-08T23:10:09.088 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:09.084344 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi077-lv-4" 2022-03-08T23:10:09.089 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:09.084353 I | op-osd: OSD PVC "smithi077-lv-4-data-0lbfgs" already exists 2022-03-08T23:10:09.089 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:09.084360 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi106-lv-1" 2022-03-08T23:10:09.089 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:09.084369 I | op-osd: OSD PVC "smithi106-lv-1-data-0j5llj" already exists 2022-03-08T23:10:09.089 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:09.084376 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi106-lv-2" 2022-03-08T23:10:09.089 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:09.084388 I | op-osd: OSD PVC "smithi106-lv-2-data-0r5dlw" already exists 2022-03-08T23:10:09.090 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:09.084397 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi106-lv-3" 2022-03-08T23:10:09.090 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:09.084407 I | op-osd: OSD PVC "smithi106-lv-3-data-07lcs6" already exists 2022-03-08T23:10:09.090 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:09.084413 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi106-lv-4" 2022-03-08T23:10:09.090 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:09.084423 I | op-osd: OSD PVC "smithi106-lv-4-data-04hgzm" already exists 2022-03-08T23:10:09.091 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:09.084433 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi135-lv-1" 2022-03-08T23:10:09.091 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:09.084468 I | op-osd: OSD PVC "smithi135-lv-1-data-0cq5dd" already exists 2022-03-08T23:10:09.091 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:09.084473 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi135-lv-2" 2022-03-08T23:10:09.091 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:09.084479 I | op-osd: OSD PVC "smithi135-lv-2-data-0jcrvg" already exists 2022-03-08T23:10:09.092 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:09.084492 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi135-lv-3" 2022-03-08T23:10:09.092 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:09.084498 I | op-osd: OSD PVC "smithi135-lv-3-data-0d9nsm" already exists 2022-03-08T23:10:09.092 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:09.084503 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi135-lv-4" 2022-03-08T23:10:09.092 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:09.084512 I | op-osd: OSD PVC "smithi135-lv-4-data-07k7pp" already exists 2022-03-08T23:10:09.092 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:09.084518 I | op-osd: creating 1 new PVCs for device set "smithi077-lv-3" 2022-03-08T23:10:09.286 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:09.284810 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.110.141.123:6789","10.98.41.230:6789","10.109.47.142:6789"]}] data:a=10.110.141.123:6789,b=10.98.41.230:6789,c=10.109.47.142:6789 mapping:{"node":{"a":{"Name":"smithi135","Hostname":"smithi135","Address":"172.21.15.135"},"b":{"Name":"smithi106","Hostname":"smithi106","Address":"172.21.15.106"},"c":{"Name":"smithi077","Hostname":"smithi077","Address":"172.21.15.77"}}} maxMonId:2] 2022-03-08T23:10:09.488 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:09.487200 I | op-osd: successfully provisioned PVC "smithi077-lv-3-data-0dl2nd" 2022-03-08T23:10:09.536 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:09.533954 I | op-osd: OSD will have its main bluestore block on "smithi077-lv-1-data-0hb6kc" 2022-03-08T23:10:09.536 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:09.533974 I | op-osd: skipping OSD prepare job creation for PVC "smithi077-lv-1-data-0hb6kc" because OSD daemon using the PVC already exists 2022-03-08T23:10:09.536 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:09.533979 I | op-osd: OSD will have its main bluestore block on "smithi077-lv-2-data-0hqxxt" 2022-03-08T23:10:09.537 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:09.533982 I | op-osd: skipping OSD prepare job creation for PVC "smithi077-lv-2-data-0hqxxt" because OSD daemon using the PVC already exists 2022-03-08T23:10:09.537 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:09.533985 I | op-osd: OSD will have its main bluestore block on "smithi077-lv-4-data-0lbfgs" 2022-03-08T23:10:09.537 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:09.533988 I | op-osd: skipping OSD prepare job creation for PVC "smithi077-lv-4-data-0lbfgs" because OSD daemon using the PVC already exists 2022-03-08T23:10:09.537 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:09.533991 I | op-osd: OSD will have its main bluestore block on "smithi106-lv-1-data-0j5llj" 2022-03-08T23:10:09.537 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:09.533994 I | op-osd: skipping OSD prepare job creation for PVC "smithi106-lv-1-data-0j5llj" because OSD daemon using the PVC already exists 2022-03-08T23:10:09.538 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:09.533997 I | op-osd: OSD will have its main bluestore block on "smithi106-lv-2-data-0r5dlw" 2022-03-08T23:10:09.538 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:09.534000 I | op-osd: skipping OSD prepare job creation for PVC "smithi106-lv-2-data-0r5dlw" because OSD daemon using the PVC already exists 2022-03-08T23:10:09.538 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:09.534003 I | op-osd: OSD will have its main bluestore block on "smithi106-lv-3-data-07lcs6" 2022-03-08T23:10:09.538 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:09.534005 I | op-osd: skipping OSD prepare job creation for PVC "smithi106-lv-3-data-07lcs6" because OSD daemon using the PVC already exists 2022-03-08T23:10:09.539 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:09.534008 I | op-osd: OSD will have its main bluestore block on "smithi106-lv-4-data-04hgzm" 2022-03-08T23:10:09.539 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:09.534011 I | op-osd: skipping OSD prepare job creation for PVC "smithi106-lv-4-data-04hgzm" because OSD daemon using the PVC already exists 2022-03-08T23:10:09.539 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:09.534014 I | op-osd: OSD will have its main bluestore block on "smithi135-lv-1-data-0cq5dd" 2022-03-08T23:10:09.539 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:09.534016 I | op-osd: skipping OSD prepare job creation for PVC "smithi135-lv-1-data-0cq5dd" because OSD daemon using the PVC already exists 2022-03-08T23:10:09.539 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:09.534019 I | op-osd: OSD will have its main bluestore block on "smithi135-lv-2-data-0jcrvg" 2022-03-08T23:10:09.540 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:09.534022 I | op-osd: skipping OSD prepare job creation for PVC "smithi135-lv-2-data-0jcrvg" because OSD daemon using the PVC already exists 2022-03-08T23:10:09.540 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:09.534025 I | op-osd: OSD will have its main bluestore block on "smithi135-lv-3-data-0d9nsm" 2022-03-08T23:10:09.540 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:09.534027 I | op-osd: skipping OSD prepare job creation for PVC "smithi135-lv-3-data-0d9nsm" because OSD daemon using the PVC already exists 2022-03-08T23:10:09.541 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:09.534030 I | op-osd: OSD will have its main bluestore block on "smithi135-lv-4-data-07k7pp" 2022-03-08T23:10:09.541 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:09.534033 I | op-osd: skipping OSD prepare job creation for PVC "smithi135-lv-4-data-07k7pp" because OSD daemon using the PVC already exists 2022-03-08T23:10:09.541 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:09.534036 I | op-osd: OSD will have its main bluestore block on "smithi077-lv-3-data-0dl2nd" 2022-03-08T23:10:10.154 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:10.153376 I | op-mgr: successful modules: prometheus 2022-03-08T23:10:10.174 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:10.173216 I | op-mgr: successful modules: mgr module(s) from the spec 2022-03-08T23:10:10.174 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:10.173263 I | op-mgr: successful modules: dashboard 2022-03-08T23:10:10.300 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:10.299507 I | op-osd: started OSD provisioning job for PVC "smithi077-lv-3-data-0dl2nd" 2022-03-08T23:10:10.300 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:10.299545 I | op-osd: start provisioning the OSDs on nodes, if needed 2022-03-08T23:10:10.301 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:10.299551 I | op-osd: no nodes are defined for configuring OSDs on raw devices 2022-03-08T23:10:10.482 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:10.481709 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-08T23:10:10.487 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:10.481992 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-08T23:10:10.753 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:10.752280 I | op-osd: OSD orchestration status for PVC smithi077-lv-3-data-0dl2nd is "starting" 2022-03-08T23:10:10.855 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:10.854122 I | op-osd: skipping osd checks for ok-to-stop 2022-03-08T23:10:10.937 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:10.936597 I | op-osd: updating OSD 1 on PVC "smithi106-lv-2-data-0r5dlw" 2022-03-08T23:10:10.937 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:10.936629 I | op-osd: OSD will have its main bluestore block on "smithi106-lv-2-data-0r5dlw" 2022-03-08T23:10:11.400 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:11.399839 I | op-mon: deployment for mon rook-ceph-mon-a already exists. updating if needed 2022-03-08T23:10:11.409 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:11.409025 I | op-k8sutil: deployment "rook-ceph-mon-a" did not change, nothing to update 2022-03-08T23:10:11.410 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:11.409050 I | op-mon: waiting for mon quorum with [a b c] 2022-03-08T23:10:11.503 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:11.501892 I | op-osd: skipping osd checks for ok-to-stop 2022-03-08T23:10:11.506 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:11.505772 I | op-osd: updating OSD 10 on PVC "smithi077-lv-2-data-0hqxxt" 2022-03-08T23:10:11.507 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:11.505795 I | op-osd: OSD will have its main bluestore block on "smithi077-lv-2-data-0hqxxt" 2022-03-08T23:10:11.692 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:11.691184 I | op-mon: mons running: [a b c] 2022-03-08T23:10:11.777 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:11.776745 I | op-osd: skipping osd checks for ok-to-stop 2022-03-08T23:10:11.885 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:11.884522 I | op-osd: updating OSD 11 on PVC "smithi106-lv-1-data-0j5llj" 2022-03-08T23:10:11.885 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:11.884571 I | op-osd: OSD will have its main bluestore block on "smithi106-lv-1-data-0j5llj" 2022-03-08T23:10:12.111 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:12.110724 I | op-mon: Monitors in quorum: [a b c] 2022-03-08T23:10:12.128 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:12.127994 I | op-mon: deployment for mon rook-ceph-mon-b already exists. updating if needed 2022-03-08T23:10:12.138 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:12.137410 I | op-k8sutil: deployment "rook-ceph-mon-b" did not change, nothing to update 2022-03-08T23:10:12.138 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:12.137441 I | op-mon: waiting for mon quorum with [a b c] 2022-03-08T23:10:12.302 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:12.301652 I | op-mon: mons running: [a b c] 2022-03-08T23:10:12.404 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:12.402823 I | op-osd: skipping osd checks for ok-to-stop 2022-03-08T23:10:12.412 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:12.411697 I | op-mgr: successful modules: orchestrator modules 2022-03-08T23:10:12.463 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:12.462023 I | op-osd: updating OSD 2 on PVC "smithi106-lv-4-data-04hgzm" 2022-03-08T23:10:12.463 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:12.462059 I | op-osd: OSD will have its main bluestore block on "smithi106-lv-4-data-04hgzm" 2022-03-08T23:10:12.707 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:12.706007 I | op-mon: Monitors in quorum: [a b c] 2022-03-08T23:10:13.186 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:13.185258 I | op-mon: deployment for mon rook-ceph-mon-c already exists. updating if needed 2022-03-08T23:10:13.367 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:13.366328 I | op-osd: skipping osd checks for ok-to-stop 2022-03-08T23:10:13.468 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:13.467048 I | op-k8sutil: deployment "rook-ceph-mon-c" did not change, nothing to update 2022-03-08T23:10:13.468 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:13.467085 I | op-mon: waiting for mon quorum with [a b c] 2022-03-08T23:10:13.494 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:13.493314 I | op-mon: mons running: [a b c] 2022-03-08T23:10:13.663 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:13.662144 I | op-osd: updating OSD 3 on PVC "smithi135-lv-3-data-0d9nsm" 2022-03-08T23:10:13.663 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:13.662195 I | op-osd: OSD will have its main bluestore block on "smithi135-lv-3-data-0d9nsm" 2022-03-08T23:10:13.947 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:13.946039 I | op-mon: Monitors in quorum: [a b c] 2022-03-08T23:10:13.947 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:13.946058 I | op-mon: mons created: 3 2022-03-08T23:10:14.370 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:14.369626 I | op-osd: skipping osd checks for ok-to-stop 2022-03-08T23:10:14.389 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:14.388448 I | op-mon: waiting for mon quorum with [a b c] 2022-03-08T23:10:14.421 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:14.420577 I | op-mon: mons running: [a b c] 2022-03-08T23:10:14.824 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:14.823179 I | op-mon: Monitors in quorum: [a b c] 2022-03-08T23:10:14.824 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:14.823225 I | cephclient: getting or creating ceph auth key "client.csi-rbd-provisioner" 2022-03-08T23:10:14.871 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:14.870595 I | op-osd: updating OSD 4 on PVC "smithi135-lv-4-data-07k7pp" 2022-03-08T23:10:14.871 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:14.870637 I | op-osd: OSD will have its main bluestore block on "smithi135-lv-4-data-07k7pp" 2022-03-08T23:10:15.418 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:15.417789 I | op-osd: skipping osd checks for ok-to-stop 2022-03-08T23:10:15.430 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:15.429803 I | cephclient: getting or creating ceph auth key "client.csi-rbd-node" 2022-03-08T23:10:15.607 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:15.606598 I | op-osd: updating OSD 5 on PVC "smithi077-lv-4-data-0lbfgs" 2022-03-08T23:10:15.607 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:15.606634 I | op-osd: OSD will have its main bluestore block on "smithi077-lv-4-data-0lbfgs" 2022-03-08T23:10:15.861 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:15.860034 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-provisioner" 2022-03-08T23:10:15.968 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:15.966747 I | op-osd: skipping osd checks for ok-to-stop 2022-03-08T23:10:16.093 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:16.091933 I | op-osd: updating OSD 6 on PVC "smithi135-lv-2-data-0jcrvg" 2022-03-08T23:10:16.093 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:16.091960 I | op-osd: OSD will have its main bluestore block on "smithi135-lv-2-data-0jcrvg" 2022-03-08T23:10:16.278 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:16.277676 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-node" 2022-03-08T23:10:16.725 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:16.724927 I | ceph-csi: created kubernetes csi secrets for cluster "rook-ceph" 2022-03-08T23:10:16.726 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:16.724957 I | cephclient: getting or creating ceph auth key "client.crash" 2022-03-08T23:10:16.776 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:16.775629 I | op-osd: skipping osd checks for ok-to-stop 2022-03-08T23:10:17.091 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:17.089787 I | op-osd: updating OSD 7 on PVC "smithi135-lv-1-data-0cq5dd" 2022-03-08T23:10:17.091 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:17.089820 I | op-osd: OSD will have its main bluestore block on "smithi135-lv-1-data-0cq5dd" 2022-03-08T23:10:17.307 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:17.305564 I | ceph-crashcollector-controller: created kubernetes crash collector secret for cluster "rook-ceph" 2022-03-08T23:10:17.569 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:17.568241 I | op-osd: skipping osd checks for ok-to-stop 2022-03-08T23:10:17.662 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:17.661439 I | op-osd: updating OSD 8 on PVC "smithi106-lv-3-data-07lcs6" 2022-03-08T23:10:17.662 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:17.661472 I | op-osd: OSD will have its main bluestore block on "smithi106-lv-3-data-07lcs6" 2022-03-08T23:10:17.725 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:17.724481 I | cephclient: successfully enabled msgr2 protocol 2022-03-08T23:10:17.725 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:17.724524 I | cephclient: create rbd-mirror bootstrap peer token "client.rbd-mirror-peer" 2022-03-08T23:10:17.726 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:17.724539 I | cephclient: getting or creating ceph auth key "client.rbd-mirror-peer" 2022-03-08T23:10:18.136 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:18.135904 I | cephclient: successfully created rbd-mirror bootstrap peer token for cluster "rook-ceph" 2022-03-08T23:10:18.189 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:18.188219 I | op-mgr: start running mgr 2022-03-08T23:10:18.189 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:18.188254 I | cephclient: getting or creating ceph auth key "mgr.a" 2022-03-08T23:10:18.469 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:18.468006 I | op-osd: skipping osd checks for ok-to-stop 2022-03-08T23:10:18.531 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:18.530534 I | op-osd: updating OSD 9 on PVC "smithi077-lv-1-data-0hb6kc" 2022-03-08T23:10:18.531 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:18.530560 I | op-osd: OSD will have its main bluestore block on "smithi077-lv-1-data-0hb6kc" 2022-03-08T23:10:18.906 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:18.905608 I | op-mgr: deployment for mgr rook-ceph-mgr-a already exists. updating if needed 2022-03-08T23:10:19.268 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:19.267784 I | op-k8sutil: deployment "rook-ceph-mgr-a" did not change, nothing to update 2022-03-08T23:10:19.467 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:19.465982 I | op-osd: OSD orchestration status for node smithi077-lv-3-data-0dl2nd is "orchestrating" 2022-03-08T23:10:19.467 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:19.466277 I | op-osd: OSD orchestration status for PVC smithi077-lv-3-data-0dl2nd is "orchestrating" 2022-03-08T23:10:19.662 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:19.661842 I | op-mgr: setting services to point to mgr "a" 2022-03-08T23:10:19.797 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:19.796437 I | op-mgr: no need to update service "rook-ceph-mgr" 2022-03-08T23:10:19.798 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:19.796580 I | op-mgr: successful modules: balancer 2022-03-08T23:10:19.810 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:19.809000 I | op-osd: start running osds in namespace "rook-ceph" 2022-03-08T23:10:19.810 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:19.809029 I | op-osd: wait timeout for healthy OSDs during upgrade or restart is "10m0s" 2022-03-08T23:10:19.912 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:19.906780 I | op-osd: start provisioning the OSDs on PVCs, if needed 2022-03-08T23:10:19.917 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:19.915370 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi077-lv-1" 2022-03-08T23:10:19.917 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:19.915405 I | op-osd: OSD PVC "smithi077-lv-1-data-0hb6kc" already exists 2022-03-08T23:10:19.918 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:19.915415 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi077-lv-2" 2022-03-08T23:10:19.918 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:19.915424 I | op-osd: OSD PVC "smithi077-lv-2-data-0hqxxt" already exists 2022-03-08T23:10:19.918 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:19.915432 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi077-lv-4" 2022-03-08T23:10:19.918 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:19.915441 I | op-osd: OSD PVC "smithi077-lv-4-data-0lbfgs" already exists 2022-03-08T23:10:19.919 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:19.915450 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi106-lv-1" 2022-03-08T23:10:19.919 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:19.915460 I | op-osd: OSD PVC "smithi106-lv-1-data-0j5llj" already exists 2022-03-08T23:10:19.919 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:19.915468 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi106-lv-2" 2022-03-08T23:10:19.919 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:19.915477 I | op-osd: OSD PVC "smithi106-lv-2-data-0r5dlw" already exists 2022-03-08T23:10:19.920 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:19.915487 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi106-lv-3" 2022-03-08T23:10:19.920 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:19.915495 I | op-osd: OSD PVC "smithi106-lv-3-data-07lcs6" already exists 2022-03-08T23:10:19.920 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:19.915504 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi106-lv-4" 2022-03-08T23:10:19.920 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:19.915513 I | op-osd: OSD PVC "smithi106-lv-4-data-04hgzm" already exists 2022-03-08T23:10:19.920 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:19.915519 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi135-lv-1" 2022-03-08T23:10:19.921 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:19.915527 I | op-osd: OSD PVC "smithi135-lv-1-data-0cq5dd" already exists 2022-03-08T23:10:19.921 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:19.915534 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi135-lv-2" 2022-03-08T23:10:19.921 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:19.915542 I | op-osd: OSD PVC "smithi135-lv-2-data-0jcrvg" already exists 2022-03-08T23:10:19.921 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:19.915549 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi135-lv-3" 2022-03-08T23:10:19.922 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:19.915560 I | op-osd: OSD PVC "smithi135-lv-3-data-0d9nsm" already exists 2022-03-08T23:10:19.922 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:19.915569 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi135-lv-4" 2022-03-08T23:10:19.922 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:19.915580 I | op-osd: OSD PVC "smithi135-lv-4-data-07k7pp" already exists 2022-03-08T23:10:19.922 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:19.915589 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi077-lv-3" 2022-03-08T23:10:19.923 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:19.915597 I | op-osd: OSD PVC "smithi077-lv-3-data-0dl2nd" already exists 2022-03-08T23:10:20.097 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:20.096100 I | op-osd: OSD will have its main bluestore block on "smithi077-lv-1-data-0hb6kc" 2022-03-08T23:10:20.098 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:20.096127 I | op-osd: skipping OSD prepare job creation for PVC "smithi077-lv-1-data-0hb6kc" because OSD daemon using the PVC already exists 2022-03-08T23:10:20.098 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:20.096134 I | op-osd: OSD will have its main bluestore block on "smithi077-lv-2-data-0hqxxt" 2022-03-08T23:10:20.098 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:20.096138 I | op-osd: skipping OSD prepare job creation for PVC "smithi077-lv-2-data-0hqxxt" because OSD daemon using the PVC already exists 2022-03-08T23:10:20.101 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:20.096143 I | op-osd: OSD will have its main bluestore block on "smithi077-lv-4-data-0lbfgs" 2022-03-08T23:10:20.101 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:20.096148 I | op-osd: skipping OSD prepare job creation for PVC "smithi077-lv-4-data-0lbfgs" because OSD daemon using the PVC already exists 2022-03-08T23:10:20.101 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:20.096153 I | op-osd: OSD will have its main bluestore block on "smithi106-lv-1-data-0j5llj" 2022-03-08T23:10:20.101 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:20.096157 I | op-osd: skipping OSD prepare job creation for PVC "smithi106-lv-1-data-0j5llj" because OSD daemon using the PVC already exists 2022-03-08T23:10:20.102 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:20.096162 I | op-osd: OSD will have its main bluestore block on "smithi106-lv-2-data-0r5dlw" 2022-03-08T23:10:20.102 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:20.096167 I | op-osd: skipping OSD prepare job creation for PVC "smithi106-lv-2-data-0r5dlw" because OSD daemon using the PVC already exists 2022-03-08T23:10:20.102 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:20.096172 I | op-osd: OSD will have its main bluestore block on "smithi106-lv-3-data-07lcs6" 2022-03-08T23:10:20.102 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:20.096176 I | op-osd: skipping OSD prepare job creation for PVC "smithi106-lv-3-data-07lcs6" because OSD daemon using the PVC already exists 2022-03-08T23:10:20.103 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:20.096182 I | op-osd: OSD will have its main bluestore block on "smithi106-lv-4-data-04hgzm" 2022-03-08T23:10:20.103 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:20.096187 I | op-osd: skipping OSD prepare job creation for PVC "smithi106-lv-4-data-04hgzm" because OSD daemon using the PVC already exists 2022-03-08T23:10:20.103 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:20.096192 I | op-osd: OSD will have its main bluestore block on "smithi135-lv-1-data-0cq5dd" 2022-03-08T23:10:20.103 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:20.096197 I | op-osd: skipping OSD prepare job creation for PVC "smithi135-lv-1-data-0cq5dd" because OSD daemon using the PVC already exists 2022-03-08T23:10:20.103 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:20.096202 I | op-osd: OSD will have its main bluestore block on "smithi135-lv-2-data-0jcrvg" 2022-03-08T23:10:20.104 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:20.096206 I | op-osd: skipping OSD prepare job creation for PVC "smithi135-lv-2-data-0jcrvg" because OSD daemon using the PVC already exists 2022-03-08T23:10:20.104 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:20.096211 I | op-osd: OSD will have its main bluestore block on "smithi135-lv-3-data-0d9nsm" 2022-03-08T23:10:20.104 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:20.096216 I | op-osd: skipping OSD prepare job creation for PVC "smithi135-lv-3-data-0d9nsm" because OSD daemon using the PVC already exists 2022-03-08T23:10:20.104 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:20.096221 I | op-osd: OSD will have its main bluestore block on "smithi135-lv-4-data-07k7pp" 2022-03-08T23:10:20.105 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:20.096227 I | op-osd: skipping OSD prepare job creation for PVC "smithi135-lv-4-data-07k7pp" because OSD daemon using the PVC already exists 2022-03-08T23:10:20.105 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:20.096232 I | op-osd: OSD will have its main bluestore block on "smithi077-lv-3-data-0dl2nd" 2022-03-08T23:10:20.115 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:20.104971 I | op-osd: OSD orchestration status for PVC smithi077-lv-3-data-0dl2nd is "starting" 2022-03-08T23:10:20.177 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:20.176108 I | op-k8sutil: Found previous job rook-ceph-osd-prepare-smithi077-lv-3-data-0dl2nd. Status={Conditions:[] StartTime:2022-03-08 23:10:10 +0000 UTC CompletionTime: Active:1 Succeeded:0 Failed:0 CompletedIndexes: UncountedTerminatedPods:&UncountedTerminatedPods{Succeeded:[],Failed:[],}} 2022-03-08T23:10:20.177 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:20.176138 I | op-osd: started OSD provisioning job for PVC "smithi077-lv-3-data-0dl2nd" 2022-03-08T23:10:20.177 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:20.176153 I | op-osd: start provisioning the OSDs on nodes, if needed 2022-03-08T23:10:20.177 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:20.176164 I | op-osd: no nodes are defined for configuring OSDs on raw devices 2022-03-08T23:10:20.186 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:20.185592 I | op-osd: OSD orchestration status for PVC smithi077-lv-3-data-0dl2nd is "starting" 2022-03-08T23:10:20.288 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:20.287039 I | op-osd: skipping osd checks for ok-to-stop 2022-03-08T23:10:20.320 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:20.319222 I | op-osd: updating OSD 1 on PVC "smithi106-lv-2-data-0r5dlw" 2022-03-08T23:10:20.320 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:20.319257 I | op-osd: OSD will have its main bluestore block on "smithi106-lv-2-data-0r5dlw" 2022-03-08T23:10:20.966 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:20.965289 I | op-osd: skipping osd checks for ok-to-stop 2022-03-08T23:10:21.062 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:21.061465 I | op-osd: updating OSD 10 on PVC "smithi077-lv-2-data-0hqxxt" 2022-03-08T23:10:21.062 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:21.061502 I | op-osd: OSD will have its main bluestore block on "smithi077-lv-2-data-0hqxxt" 2022-03-08T23:10:21.544 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:21.543188 I | op-mgr: successful modules: prometheus 2022-03-08T23:10:21.551 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:21.550139 I | op-mgr: successful modules: mgr module(s) from the spec 2022-03-08T23:10:21.554 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:21.550632 I | op-mgr: successful modules: dashboard 2022-03-08T23:10:21.769 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:21.768324 I | op-osd: skipping osd checks for ok-to-stop 2022-03-08T23:10:21.860 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:21.859868 I | op-osd: updating OSD 11 on PVC "smithi106-lv-1-data-0j5llj" 2022-03-08T23:10:21.861 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:21.859898 I | op-osd: OSD will have its main bluestore block on "smithi106-lv-1-data-0j5llj" 2022-03-08T23:10:22.574 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:22.573338 I | op-osd: skipping osd checks for ok-to-stop 2022-03-08T23:10:22.662 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:22.661754 I | op-osd: updating OSD 2 on PVC "smithi106-lv-4-data-04hgzm" 2022-03-08T23:10:22.663 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:22.661799 I | op-osd: OSD will have its main bluestore block on "smithi106-lv-4-data-04hgzm" 2022-03-08T23:10:23.126 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:23.125734 I | op-mgr: successful modules: orchestrator modules 2022-03-08T23:10:23.369 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:23.368261 I | op-osd: skipping osd checks for ok-to-stop 2022-03-08T23:10:23.462 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:23.461117 I | op-osd: updating OSD 3 on PVC "smithi135-lv-3-data-0d9nsm" 2022-03-08T23:10:23.462 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:23.461151 I | op-osd: OSD will have its main bluestore block on "smithi135-lv-3-data-0d9nsm" 2022-03-08T23:10:24.170 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:24.169825 I | op-osd: skipping osd checks for ok-to-stop 2022-03-08T23:10:24.262 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:24.261572 I | op-osd: updating OSD 4 on PVC "smithi135-lv-4-data-07k7pp" 2022-03-08T23:10:24.263 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:24.261606 I | op-osd: OSD will have its main bluestore block on "smithi135-lv-4-data-07k7pp" 2022-03-08T23:10:24.967 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:24.966296 I | op-osd: skipping osd checks for ok-to-stop 2022-03-08T23:10:25.041 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:25.040336 I | op-osd: OSD orchestration status for PVC smithi077-lv-3-data-0dl2nd is "completed" 2022-03-08T23:10:25.041 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:25.040374 I | op-osd: creating OSD 0 on PVC "smithi077-lv-3-data-0dl2nd" 2022-03-08T23:10:25.041 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:25.040399 I | op-osd: OSD will have its main bluestore block on "smithi077-lv-3-data-0dl2nd" 2022-03-08T23:10:25.127 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:25.126210 I | op-osd: updating OSD 5 on PVC "smithi077-lv-4-data-0lbfgs" 2022-03-08T23:10:25.127 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:25.126245 I | op-osd: OSD will have its main bluestore block on "smithi077-lv-4-data-0lbfgs" 2022-03-08T23:10:25.866 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:25.864934 I | op-osd: OSD orchestration status for PVC smithi077-lv-3-data-0dl2nd is "completed" 2022-03-08T23:10:25.866 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:25.864967 I | op-osd: creating OSD 0 on PVC "smithi077-lv-3-data-0dl2nd" 2022-03-08T23:10:25.866 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:25.864981 I | op-osd: OSD will have its main bluestore block on "smithi077-lv-3-data-0dl2nd" 2022-03-08T23:10:26.597 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:26.596882 E | op-osd: failed to create OSD 0 on PVC "smithi077-lv-3-data-0dl2nd": failed to create deployment for OSD 0 on PVC "smithi077-lv-3-data-0dl2nd": deployments.apps "rook-ceph-osd-0" already exists 2022-03-08T23:10:26.661 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:26.659814 I | op-osd: finished running OSDs in namespace "rook-ceph" 2022-03-08T23:10:26.661 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:26.659842 I | ceph-cluster-controller: done reconciling ceph cluster in namespace "rook-ceph" 2022-03-08T23:10:26.705 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:26.704551 I | op-osd: skipping osd checks for ok-to-stop 2022-03-08T23:10:26.766 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:26.764944 I | op-osd: updating OSD 6 on PVC "smithi135-lv-2-data-0jcrvg" 2022-03-08T23:10:26.766 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:26.764981 I | op-osd: OSD will have its main bluestore block on "smithi135-lv-2-data-0jcrvg" 2022-03-08T23:10:27.446 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:27.445474 I | op-osd: skipping osd checks for ok-to-stop 2022-03-08T23:10:27.547 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:27.546661 I | op-osd: updating OSD 7 on PVC "smithi135-lv-1-data-0cq5dd" 2022-03-08T23:10:27.548 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:27.546697 I | op-osd: OSD will have its main bluestore block on "smithi135-lv-1-data-0cq5dd" 2022-03-08T23:10:28.270 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:28.269947 I | op-osd: skipping osd checks for ok-to-stop 2022-03-08T23:10:28.275 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:28.274923 I | op-osd: updating OSD 8 on PVC "smithi106-lv-3-data-07lcs6" 2022-03-08T23:10:28.275 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:28.274965 I | op-osd: OSD will have its main bluestore block on "smithi106-lv-3-data-07lcs6" 2022-03-08T23:10:28.971 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:28.970605 I | op-osd: skipping osd checks for ok-to-stop 2022-03-08T23:10:29.059 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:29.059078 I | op-osd: updating OSD 9 on PVC "smithi077-lv-1-data-0hb6kc" 2022-03-08T23:10:29.060 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:29.059110 I | op-osd: OSD will have its main bluestore block on "smithi077-lv-1-data-0hb6kc" 2022-03-08T23:10:29.696 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:29.695794 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". failed to reconcile cluster "rook-ceph": failed to configure local ceph cluster: failed to create cluster: failed to start ceph osds: 1 failures encountered while running osds on nodes in namespace "rook-ceph". 2022-03-08T23:10:29.697 INFO:tasks.rook.operator.smithi077.stdout:failed to create OSD 0 on PVC "smithi077-lv-3-data-0dl2nd": failed to create deployment for OSD 0 on PVC "smithi077-lv-3-data-0dl2nd": deployments.apps "rook-ceph-osd-0" already exists 2022-03-08T23:10:29.705 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:29.701771 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-03-08T23:10:29.941 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:29.940151 I | op-mon: parsing mon endpoints: a=10.110.141.123:6789,b=10.98.41.230:6789,c=10.109.47.142:6789 2022-03-08T23:10:29.965 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:29.964600 I | ceph-spec: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3... 2022-03-08T23:10:36.146 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:36.144986 I | ceph-spec: detected ceph image version: "17.0.0-11025 quincy" 2022-03-08T23:10:36.146 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:36.145012 I | ceph-cluster-controller: validating ceph version from provided image 2022-03-08T23:10:36.146 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:36.145025 W | ceph-cluster-controller: unsupported ceph version detected: "17.0.0-11025 quincy", pursuing 2022-03-08T23:10:36.201 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:36.200292 I | op-mon: parsing mon endpoints: a=10.110.141.123:6789,b=10.98.41.230:6789,c=10.109.47.142:6789 2022-03-08T23:10:36.364 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:36.363182 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-08T23:10:36.366 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:36.363402 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-08T23:10:36.518 INFO:teuthology.orchestra.run.smithi077.stderr:+ '[' 11 '!=' 12 ']' 2022-03-08T23:10:36.519 INFO:teuthology.orchestra.run.smithi077.stderr:+ echo 'waiting for osd to come back up' 2022-03-08T23:10:36.519 INFO:teuthology.orchestra.run.smithi077.stdout:waiting for osd to come back up 2022-03-08T23:10:36.520 INFO:teuthology.orchestra.run.smithi077.stderr:++ toolbox ceph osd stat 2022-03-08T23:10:36.520 INFO:teuthology.orchestra.run.smithi077.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-03-08T23:10:36.520 INFO:teuthology.orchestra.run.smithi077.stderr:++ cut -f3 -d ' ' 2022-03-08T23:10:36.619 INFO:teuthology.orchestra.run.smithi077.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-08T23:10:36.874 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:36.873888 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-11025 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3" 2022-03-08T23:10:37.114 INFO:teuthology.orchestra.run.smithi077.stderr:+ num_osd=11 2022-03-08T23:10:37.114 INFO:teuthology.orchestra.run.smithi077.stderr:+ sleep 30s 2022-03-08T23:10:37.131 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:37.130947 I | op-mon: start running mons 2022-03-08T23:10:37.158 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:37.157820 I | op-mon: parsing mon endpoints: a=10.110.141.123:6789,b=10.98.41.230:6789,c=10.109.47.142:6789 2022-03-08T23:10:37.210 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:37.209900 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.110.141.123:6789","10.98.41.230:6789","10.109.47.142:6789"]}] data:a=10.110.141.123:6789,b=10.98.41.230:6789,c=10.109.47.142:6789 mapping:{"node":{"a":{"Name":"smithi135","Hostname":"smithi135","Address":"172.21.15.135"},"b":{"Name":"smithi106","Hostname":"smithi106","Address":"172.21.15.106"},"c":{"Name":"smithi077","Hostname":"smithi077","Address":"172.21.15.77"}}} maxMonId:2] 2022-03-08T23:10:37.273 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:37.272537 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-08T23:10:37.472 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:37.471089 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-08T23:10:38.965 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:38.964229 I | op-mon: targeting the mon count 3 2022-03-08T23:10:38.971 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:38.971466 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-08T23:10:39.327 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:39.326990 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-08T23:10:39.328 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:39.327020 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-03-08T23:10:39.664 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:39.663591 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-03-08T23:10:39.664 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:39.663622 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-08T23:10:40.010 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:40.009673 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-08T23:10:40.010 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:40.009697 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-03-08T23:10:40.552 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:40.551542 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-03-08T23:10:40.553 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:40.551573 I | op-config: deleting "log file" option from the mon configuration database 2022-03-08T23:10:40.892 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:40.891597 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-03-08T23:10:40.893 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:40.891616 I | op-mon: checking for basic quorum with existing mons 2022-03-08T23:10:41.018 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:41.017306 I | op-mon: mon "a" endpoint is [v2:10.110.141.123:3300,v1:10.110.141.123:6789] 2022-03-08T23:10:41.139 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:41.138450 I | op-mon: mon "b" endpoint is [v2:10.98.41.230:3300,v1:10.98.41.230:6789] 2022-03-08T23:10:41.299 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:41.298607 I | op-mon: mon "c" endpoint is [v2:10.109.47.142:3300,v1:10.109.47.142:6789] 2022-03-08T23:10:41.898 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:41.897954 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.98.41.230:6789","10.109.47.142:6789","10.110.141.123:6789"]}] data:a=10.110.141.123:6789,b=10.98.41.230:6789,c=10.109.47.142:6789 mapping:{"node":{"a":{"Name":"smithi135","Hostname":"smithi135","Address":"172.21.15.135"},"b":{"Name":"smithi106","Hostname":"smithi106","Address":"172.21.15.106"},"c":{"Name":"smithi077","Hostname":"smithi077","Address":"172.21.15.77"}}} maxMonId:2] 2022-03-08T23:10:42.497 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:42.496077 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-08T23:10:42.497 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:42.496346 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-08T23:10:42.915 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:42.914789 I | op-mon: deployment for mon rook-ceph-mon-a already exists. updating if needed 2022-03-08T23:10:42.924 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:42.924429 I | op-k8sutil: deployment "rook-ceph-mon-a" did not change, nothing to update 2022-03-08T23:10:42.925 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:42.924457 I | op-mon: waiting for mon quorum with [a b c] 2022-03-08T23:10:43.506 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:43.506031 I | op-mon: mons running: [a b c] 2022-03-08T23:10:44.010 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:44.009972 I | op-mon: Monitors in quorum: [a b c] 2022-03-08T23:10:44.017 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:44.016613 I | op-mon: deployment for mon rook-ceph-mon-b already exists. updating if needed 2022-03-08T23:10:44.024 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:44.023881 I | op-k8sutil: deployment "rook-ceph-mon-b" did not change, nothing to update 2022-03-08T23:10:44.024 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:44.023902 I | op-mon: waiting for mon quorum with [a b c] 2022-03-08T23:10:44.104 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:44.103822 I | op-mon: mons running: [a b c] 2022-03-08T23:10:44.540 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:44.539298 I | op-mon: Monitors in quorum: [a b c] 2022-03-08T23:10:44.550 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:44.549610 I | op-mon: deployment for mon rook-ceph-mon-c already exists. updating if needed 2022-03-08T23:10:44.557 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:44.556483 I | op-k8sutil: deployment "rook-ceph-mon-c" did not change, nothing to update 2022-03-08T23:10:44.557 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:44.556500 I | op-mon: waiting for mon quorum with [a b c] 2022-03-08T23:10:44.707 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:44.707047 I | op-mon: mons running: [a b c] 2022-03-08T23:10:45.121 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:45.120959 I | op-mon: Monitors in quorum: [a b c] 2022-03-08T23:10:45.122 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:45.120986 I | op-mon: mons created: 3 2022-03-08T23:10:45.520 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:45.519604 I | op-mon: waiting for mon quorum with [a b c] 2022-03-08T23:10:45.588 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:45.587598 I | op-mon: mons running: [a b c] 2022-03-08T23:10:45.998 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:45.997689 I | op-mon: Monitors in quorum: [a b c] 2022-03-08T23:10:45.999 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:45.997747 I | cephclient: getting or creating ceph auth key "client.csi-rbd-provisioner" 2022-03-08T23:10:46.468 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:46.467940 I | cephclient: getting or creating ceph auth key "client.csi-rbd-node" 2022-03-08T23:10:46.885 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:46.884181 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-provisioner" 2022-03-08T23:10:47.310 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:47.309097 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-node" 2022-03-08T23:10:47.765 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:47.764500 I | ceph-csi: created kubernetes csi secrets for cluster "rook-ceph" 2022-03-08T23:10:47.765 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:47.764534 I | cephclient: getting or creating ceph auth key "client.crash" 2022-03-08T23:10:48.189 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:48.188485 I | ceph-crashcollector-controller: created kubernetes crash collector secret for cluster "rook-ceph" 2022-03-08T23:10:48.660 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:48.659446 I | cephclient: successfully enabled msgr2 protocol 2022-03-08T23:10:48.660 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:48.659486 I | cephclient: create rbd-mirror bootstrap peer token "client.rbd-mirror-peer" 2022-03-08T23:10:48.661 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:48.659496 I | cephclient: getting or creating ceph auth key "client.rbd-mirror-peer" 2022-03-08T23:10:49.170 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:49.169220 I | cephclient: successfully created rbd-mirror bootstrap peer token for cluster "rook-ceph" 2022-03-08T23:10:49.207 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:49.207211 I | op-mgr: start running mgr 2022-03-08T23:10:49.208 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:49.207250 I | cephclient: getting or creating ceph auth key "mgr.a" 2022-03-08T23:10:49.675 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:49.674480 I | op-mgr: deployment for mgr rook-ceph-mgr-a already exists. updating if needed 2022-03-08T23:10:49.684 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:49.684249 I | op-k8sutil: deployment "rook-ceph-mgr-a" did not change, nothing to update 2022-03-08T23:10:49.688 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:49.687880 I | op-mgr: setting services to point to mgr "a" 2022-03-08T23:10:49.798 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:49.797621 I | op-mgr: no need to update service "rook-ceph-mgr" 2022-03-08T23:10:49.804 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:49.797753 I | op-mgr: successful modules: balancer 2022-03-08T23:10:49.811 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:49.810871 I | op-osd: start running osds in namespace "rook-ceph" 2022-03-08T23:10:49.811 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:49.810902 I | op-osd: wait timeout for healthy OSDs during upgrade or restart is "10m0s" 2022-03-08T23:10:49.862 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:49.861610 I | op-osd: start provisioning the OSDs on PVCs, if needed 2022-03-08T23:10:49.940 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:49.940286 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi077-lv-1" 2022-03-08T23:10:49.947 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:49.940945 I | op-osd: OSD PVC "smithi077-lv-1-data-0hb6kc" already exists 2022-03-08T23:10:49.948 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:49.940997 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi077-lv-2" 2022-03-08T23:10:49.948 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:49.941037 I | op-osd: OSD PVC "smithi077-lv-2-data-0hqxxt" already exists 2022-03-08T23:10:49.948 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:49.941071 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi077-lv-4" 2022-03-08T23:10:49.949 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:49.941112 I | op-osd: OSD PVC "smithi077-lv-4-data-0lbfgs" already exists 2022-03-08T23:10:49.949 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:49.941141 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi106-lv-1" 2022-03-08T23:10:49.949 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:49.941185 I | op-osd: OSD PVC "smithi106-lv-1-data-0j5llj" already exists 2022-03-08T23:10:49.949 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:49.941217 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi106-lv-2" 2022-03-08T23:10:49.950 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:49.941245 I | op-osd: OSD PVC "smithi106-lv-2-data-0r5dlw" already exists 2022-03-08T23:10:49.950 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:49.941277 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi106-lv-3" 2022-03-08T23:10:49.950 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:49.941328 I | op-osd: OSD PVC "smithi106-lv-3-data-07lcs6" already exists 2022-03-08T23:10:49.950 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:49.941353 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi106-lv-4" 2022-03-08T23:10:49.950 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:49.941390 I | op-osd: OSD PVC "smithi106-lv-4-data-04hgzm" already exists 2022-03-08T23:10:49.951 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:49.941442 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi135-lv-1" 2022-03-08T23:10:49.951 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:49.941471 I | op-osd: OSD PVC "smithi135-lv-1-data-0cq5dd" already exists 2022-03-08T23:10:49.951 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:49.941507 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi135-lv-2" 2022-03-08T23:10:49.951 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:49.941541 I | op-osd: OSD PVC "smithi135-lv-2-data-0jcrvg" already exists 2022-03-08T23:10:49.952 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:49.941575 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi135-lv-3" 2022-03-08T23:10:49.952 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:49.941620 I | op-osd: OSD PVC "smithi135-lv-3-data-0d9nsm" already exists 2022-03-08T23:10:49.952 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:49.941651 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi135-lv-4" 2022-03-08T23:10:49.952 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:49.941688 I | op-osd: OSD PVC "smithi135-lv-4-data-07k7pp" already exists 2022-03-08T23:10:49.952 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:49.941724 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi077-lv-3" 2022-03-08T23:10:49.953 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:49.941758 I | op-osd: OSD PVC "smithi077-lv-3-data-0dl2nd" already exists 2022-03-08T23:10:49.974 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:49.971765 I | op-osd: OSD will have its main bluestore block on "smithi077-lv-1-data-0hb6kc" 2022-03-08T23:10:49.975 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:49.971785 I | op-osd: skipping OSD prepare job creation for PVC "smithi077-lv-1-data-0hb6kc" because OSD daemon using the PVC already exists 2022-03-08T23:10:49.975 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:49.971789 I | op-osd: OSD will have its main bluestore block on "smithi077-lv-2-data-0hqxxt" 2022-03-08T23:10:49.975 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:49.971792 I | op-osd: skipping OSD prepare job creation for PVC "smithi077-lv-2-data-0hqxxt" because OSD daemon using the PVC already exists 2022-03-08T23:10:49.975 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:49.971795 I | op-osd: OSD will have its main bluestore block on "smithi077-lv-4-data-0lbfgs" 2022-03-08T23:10:49.976 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:49.971797 I | op-osd: skipping OSD prepare job creation for PVC "smithi077-lv-4-data-0lbfgs" because OSD daemon using the PVC already exists 2022-03-08T23:10:49.976 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:49.971801 I | op-osd: OSD will have its main bluestore block on "smithi106-lv-1-data-0j5llj" 2022-03-08T23:10:49.976 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:49.971804 I | op-osd: skipping OSD prepare job creation for PVC "smithi106-lv-1-data-0j5llj" because OSD daemon using the PVC already exists 2022-03-08T23:10:49.976 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:49.971807 I | op-osd: OSD will have its main bluestore block on "smithi106-lv-2-data-0r5dlw" 2022-03-08T23:10:49.976 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:49.971810 I | op-osd: skipping OSD prepare job creation for PVC "smithi106-lv-2-data-0r5dlw" because OSD daemon using the PVC already exists 2022-03-08T23:10:49.977 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:49.971814 I | op-osd: OSD will have its main bluestore block on "smithi106-lv-3-data-07lcs6" 2022-03-08T23:10:49.977 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:49.971818 I | op-osd: skipping OSD prepare job creation for PVC "smithi106-lv-3-data-07lcs6" because OSD daemon using the PVC already exists 2022-03-08T23:10:49.977 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:49.971823 I | op-osd: OSD will have its main bluestore block on "smithi106-lv-4-data-04hgzm" 2022-03-08T23:10:49.977 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:49.971827 I | op-osd: skipping OSD prepare job creation for PVC "smithi106-lv-4-data-04hgzm" because OSD daemon using the PVC already exists 2022-03-08T23:10:49.978 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:49.971833 I | op-osd: OSD will have its main bluestore block on "smithi135-lv-1-data-0cq5dd" 2022-03-08T23:10:49.978 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:49.971837 I | op-osd: skipping OSD prepare job creation for PVC "smithi135-lv-1-data-0cq5dd" because OSD daemon using the PVC already exists 2022-03-08T23:10:49.978 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:49.971842 I | op-osd: OSD will have its main bluestore block on "smithi135-lv-2-data-0jcrvg" 2022-03-08T23:10:49.978 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:49.971846 I | op-osd: skipping OSD prepare job creation for PVC "smithi135-lv-2-data-0jcrvg" because OSD daemon using the PVC already exists 2022-03-08T23:10:49.978 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:49.971851 I | op-osd: OSD will have its main bluestore block on "smithi135-lv-3-data-0d9nsm" 2022-03-08T23:10:49.979 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:49.971855 I | op-osd: skipping OSD prepare job creation for PVC "smithi135-lv-3-data-0d9nsm" because OSD daemon using the PVC already exists 2022-03-08T23:10:49.979 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:49.971861 I | op-osd: OSD will have its main bluestore block on "smithi135-lv-4-data-07k7pp" 2022-03-08T23:10:49.979 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:49.971865 I | op-osd: skipping OSD prepare job creation for PVC "smithi135-lv-4-data-07k7pp" because OSD daemon using the PVC already exists 2022-03-08T23:10:49.979 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:49.971870 I | op-osd: OSD will have its main bluestore block on "smithi077-lv-3-data-0dl2nd" 2022-03-08T23:10:49.979 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:49.971881 I | op-osd: skipping OSD prepare job creation for PVC "smithi077-lv-3-data-0dl2nd" because OSD daemon using the PVC already exists 2022-03-08T23:10:49.980 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:49.971885 I | op-osd: start provisioning the OSDs on nodes, if needed 2022-03-08T23:10:49.980 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:49.971888 I | op-osd: no nodes are defined for configuring OSDs on raw devices 2022-03-08T23:10:50.241 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:50.240867 I | op-osd: skipping osd checks for ok-to-stop 2022-03-08T23:10:50.246 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:50.245823 I | op-osd: updating OSD 0 on PVC "smithi077-lv-3-data-0dl2nd" 2022-03-08T23:10:50.246 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:50.245846 I | op-osd: OSD will have its main bluestore block on "smithi077-lv-3-data-0dl2nd" 2022-03-08T23:10:50.450 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:50.449403 I | op-osd: skipping osd checks for ok-to-stop 2022-03-08T23:10:50.456 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:50.456199 I | op-osd: updating OSD 1 on PVC "smithi106-lv-2-data-0r5dlw" 2022-03-08T23:10:50.457 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:50.456228 I | op-osd: OSD will have its main bluestore block on "smithi106-lv-2-data-0r5dlw" 2022-03-08T23:10:50.624 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:50.624052 I | op-osd: skipping osd checks for ok-to-stop 2022-03-08T23:10:50.631 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:50.630574 I | op-osd: updating OSD 10 on PVC "smithi077-lv-2-data-0hqxxt" 2022-03-08T23:10:50.631 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:50.630603 I | op-osd: OSD will have its main bluestore block on "smithi077-lv-2-data-0hqxxt" 2022-03-08T23:10:51.157 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:51.156534 I | op-osd: skipping osd checks for ok-to-stop 2022-03-08T23:10:51.249 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:51.249178 I | op-osd: updating OSD 11 on PVC "smithi106-lv-1-data-0j5llj" 2022-03-08T23:10:51.250 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:51.249205 I | op-osd: OSD will have its main bluestore block on "smithi106-lv-1-data-0j5llj" 2022-03-08T23:10:51.377 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:51.376654 I | op-mgr: successful modules: mgr module(s) from the spec 2022-03-08T23:10:51.384 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:51.383443 I | op-mgr: successful modules: dashboard 2022-03-08T23:10:51.387 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:51.383823 I | op-mgr: successful modules: prometheus 2022-03-08T23:10:51.957 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:51.956470 I | op-osd: skipping osd checks for ok-to-stop 2022-03-08T23:10:52.050 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:52.049947 I | op-osd: updating OSD 2 on PVC "smithi106-lv-4-data-04hgzm" 2022-03-08T23:10:52.051 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:52.049979 I | op-osd: OSD will have its main bluestore block on "smithi106-lv-4-data-04hgzm" 2022-03-08T23:10:52.755 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:52.754126 I | op-osd: skipping osd checks for ok-to-stop 2022-03-08T23:10:52.849 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:52.848898 I | op-osd: updating OSD 3 on PVC "smithi135-lv-3-data-0d9nsm" 2022-03-08T23:10:52.849 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:52.848938 I | op-osd: OSD will have its main bluestore block on "smithi135-lv-3-data-0d9nsm" 2022-03-08T23:10:53.043 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:53.042252 I | op-mgr: successful modules: orchestrator modules 2022-03-08T23:10:53.553 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:53.552323 I | op-osd: skipping osd checks for ok-to-stop 2022-03-08T23:10:53.649 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:53.648347 I | op-osd: updating OSD 4 on PVC "smithi135-lv-4-data-07k7pp" 2022-03-08T23:10:53.649 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:53.648372 I | op-osd: OSD will have its main bluestore block on "smithi135-lv-4-data-07k7pp" 2022-03-08T23:10:54.353 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:54.353060 I | op-osd: skipping osd checks for ok-to-stop 2022-03-08T23:10:54.449 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:54.448659 I | op-osd: updating OSD 5 on PVC "smithi077-lv-4-data-0lbfgs" 2022-03-08T23:10:54.449 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:54.448694 I | op-osd: OSD will have its main bluestore block on "smithi077-lv-4-data-0lbfgs" 2022-03-08T23:10:55.156 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:55.154986 I | op-osd: skipping osd checks for ok-to-stop 2022-03-08T23:10:55.249 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:55.249136 I | op-osd: updating OSD 6 on PVC "smithi135-lv-2-data-0jcrvg" 2022-03-08T23:10:55.250 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:55.249178 I | op-osd: OSD will have its main bluestore block on "smithi135-lv-2-data-0jcrvg" 2022-03-08T23:10:55.956 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:55.955677 I | op-osd: skipping osd checks for ok-to-stop 2022-03-08T23:10:56.049 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:56.048749 I | op-osd: updating OSD 7 on PVC "smithi135-lv-1-data-0cq5dd" 2022-03-08T23:10:56.050 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:56.048782 I | op-osd: OSD will have its main bluestore block on "smithi135-lv-1-data-0cq5dd" 2022-03-08T23:10:56.755 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:56.754866 I | op-osd: skipping osd checks for ok-to-stop 2022-03-08T23:10:56.850 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:56.849247 I | op-osd: updating OSD 8 on PVC "smithi106-lv-3-data-07lcs6" 2022-03-08T23:10:56.850 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:56.849281 I | op-osd: OSD will have its main bluestore block on "smithi106-lv-3-data-07lcs6" 2022-03-08T23:10:57.555 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:57.554202 I | op-osd: skipping osd checks for ok-to-stop 2022-03-08T23:10:57.650 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:57.650139 I | op-osd: updating OSD 9 on PVC "smithi077-lv-1-data-0hb6kc" 2022-03-08T23:10:57.651 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:57.650177 I | op-osd: OSD will have its main bluestore block on "smithi077-lv-1-data-0hb6kc" 2022-03-08T23:10:58.737 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:58.736816 I | op-osd: finished running OSDs in namespace "rook-ceph" 2022-03-08T23:10:58.738 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:10:58.736850 I | ceph-cluster-controller: done reconciling ceph cluster in namespace "rook-ceph" 2022-03-08T23:11:07.115 INFO:teuthology.orchestra.run.smithi077.stderr:+ '[' 11 '!=' 12 ']' 2022-03-08T23:11:07.115 INFO:teuthology.orchestra.run.smithi077.stdout:waiting for osd to come back up 2022-03-08T23:11:07.116 INFO:teuthology.orchestra.run.smithi077.stderr:+ echo 'waiting for osd to come back up' 2022-03-08T23:11:07.116 INFO:teuthology.orchestra.run.smithi077.stderr:++ toolbox ceph osd stat 2022-03-08T23:11:07.116 INFO:teuthology.orchestra.run.smithi077.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-03-08T23:11:07.117 INFO:teuthology.orchestra.run.smithi077.stderr:++ cut -f3 -d ' ' 2022-03-08T23:11:07.246 INFO:teuthology.orchestra.run.smithi077.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-08T23:11:07.779 INFO:teuthology.orchestra.run.smithi077.stderr:+ num_osd=12 2022-03-08T23:11:07.779 INFO:teuthology.orchestra.run.smithi077.stderr:+ sleep 30s 2022-03-08T23:11:37.780 INFO:teuthology.orchestra.run.smithi077.stderr:+ '[' 12 '!=' 12 ']' 2022-03-08T23:11:37.782 INFO:teuthology.run_tasks:Running task rook.shell... 2022-03-08T23:11:37.795 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph orch status 2022-03-08T23:11:41.034 INFO:teuthology.orchestra.run.smithi077.stdout:Backend: rook 2022-03-08T23:11:41.034 INFO:teuthology.orchestra.run.smithi077.stdout:Available: Yes 2022-03-08T23:11:41.054 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph orch ps 2022-03-08T23:11:41.581 INFO:teuthology.orchestra.run.smithi077.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID 2022-03-08T23:11:41.581 INFO:teuthology.orchestra.run.smithi077.stdout:crashcollector.crash smithi077 running (5m) 0s ago 5m - - 4b871cde5e1a 2022-03-08T23:11:41.581 INFO:teuthology.orchestra.run.smithi077.stdout:crashcollector.crash smithi106 running (4m) 0s ago 4m - - 4b871cde5e1a 2022-03-08T23:11:41.582 INFO:teuthology.orchestra.run.smithi077.stdout:crashcollector.crash smithi135 running (5m) 0s ago 5m - - 4b871cde5e1a 2022-03-08T23:11:41.582 INFO:teuthology.orchestra.run.smithi077.stdout:mgr.a smithi106 running (5m) 0s ago 5m - - 4b871cde5e1a 2022-03-08T23:11:41.582 INFO:teuthology.orchestra.run.smithi077.stdout:mon.a smithi135 running (8m) 0s ago 8m - - 4b871cde5e1a 2022-03-08T23:11:41.582 INFO:teuthology.orchestra.run.smithi077.stdout:mon.b smithi106 running (7m) 0s ago 7m - - 4b871cde5e1a 2022-03-08T23:11:41.583 INFO:teuthology.orchestra.run.smithi077.stdout:mon.c smithi077 running (6m) 0s ago 6m - - 4b871cde5e1a 2022-03-08T23:11:41.583 INFO:teuthology.orchestra.run.smithi077.stdout:osd.0 smithi077 running (75s) 0s ago 75s - - 4b871cde5e1a 2022-03-08T23:11:41.583 INFO:teuthology.orchestra.run.smithi077.stdout:osd.1 smithi106 running (4m) 0s ago 4m - - 4b871cde5e1a 2022-03-08T23:11:41.583 INFO:teuthology.orchestra.run.smithi077.stdout:osd.10 smithi077 running (3m) 0s ago 3m - - 4b871cde5e1a 2022-03-08T23:11:41.584 INFO:teuthology.orchestra.run.smithi077.stdout:osd.11 smithi106 running (3m) 0s ago 3m - - 4b871cde5e1a 2022-03-08T23:11:41.584 INFO:teuthology.orchestra.run.smithi077.stdout:osd.2 smithi106 running (4m) 0s ago 4m - - 4b871cde5e1a 2022-03-08T23:11:41.584 INFO:teuthology.orchestra.run.smithi077.stdout:osd.3 smithi135 running (4m) 0s ago 4m - - 4b871cde5e1a 2022-03-08T23:11:41.584 INFO:teuthology.orchestra.run.smithi077.stdout:osd.4 smithi135 running (4m) 0s ago 4m - - 4b871cde5e1a 2022-03-08T23:11:41.585 INFO:teuthology.orchestra.run.smithi077.stdout:osd.5 smithi077 running (4m) 0s ago 4m - - 4b871cde5e1a 2022-03-08T23:11:41.585 INFO:teuthology.orchestra.run.smithi077.stdout:osd.6 smithi135 running (4m) 0s ago 4m - - 4b871cde5e1a 2022-03-08T23:11:41.585 INFO:teuthology.orchestra.run.smithi077.stdout:osd.7 smithi135 running (4m) 0s ago 4m - - 4b871cde5e1a 2022-03-08T23:11:41.585 INFO:teuthology.orchestra.run.smithi077.stdout:osd.8 smithi106 running (4m) 0s ago 4m - - 4b871cde5e1a 2022-03-08T23:11:41.585 INFO:teuthology.orchestra.run.smithi077.stdout:osd.9 smithi077 running (3m) 0s ago 3m - - 4b871cde5e1a 2022-03-08T23:11:41.604 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph orch ls 2022-03-08T23:11:42.305 INFO:teuthology.orchestra.run.smithi077.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2022-03-08T23:11:42.306 INFO:teuthology.orchestra.run.smithi077.stdout:crash 3/3 0s ago 5m * 2022-03-08T23:11:42.306 INFO:teuthology.orchestra.run.smithi077.stdout:mgr 1/1 0s ago 5m count:1 2022-03-08T23:11:42.306 INFO:teuthology.orchestra.run.smithi077.stdout:mon 3/3 0s ago 8m count:3 2022-03-08T23:11:42.307 INFO:teuthology.orchestra.run.smithi077.stdout:osd 24 0s ago 4m 2022-03-08T23:11:42.307 INFO:teuthology.orchestra.run.smithi077.stdout:osd.all-available-devices 0 0s ago - * 2022-03-08T23:11:42.325 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph orch host ls 2022-03-08T23:11:42.859 INFO:teuthology.orchestra.run.smithi077.stdout:HOST ADDR LABELS STATUS 2022-03-08T23:11:42.860 INFO:teuthology.orchestra.run.smithi077.stdout:smithi077 172.21.15.77/smithi077 2022-03-08T23:11:42.860 INFO:teuthology.orchestra.run.smithi077.stdout:smithi106 172.21.15.106/smithi106 2022-03-08T23:11:42.860 INFO:teuthology.orchestra.run.smithi077.stdout:smithi135 172.21.15.135/smithi135 2022-03-08T23:11:42.861 INFO:teuthology.orchestra.run.smithi077.stdout:3 hosts in cluster 2022-03-08T23:11:42.878 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph orch device ls 2022-03-08T23:11:43.435 INFO:teuthology.orchestra.run.smithi077.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2022-03-08T23:11:43.436 INFO:teuthology.orchestra.run.smithi077.stdout:smithi077 /dev/vg_nvme/lv_1 unknown None 107G 0s ago 2022-03-08T23:11:43.436 INFO:teuthology.orchestra.run.smithi077.stdout:smithi077 /dev/vg_nvme/lv_2 unknown None 107G 0s ago 2022-03-08T23:11:43.436 INFO:teuthology.orchestra.run.smithi077.stdout:smithi077 /dev/vg_nvme/lv_3 unknown None 107G 0s ago 2022-03-08T23:11:43.436 INFO:teuthology.orchestra.run.smithi077.stdout:smithi077 /dev/vg_nvme/lv_4 unknown None 107G 0s ago 2022-03-08T23:11:43.437 INFO:teuthology.orchestra.run.smithi077.stdout:smithi106 /dev/vg_nvme/lv_1 unknown None 107G 0s ago 2022-03-08T23:11:43.437 INFO:teuthology.orchestra.run.smithi077.stdout:smithi106 /dev/vg_nvme/lv_2 unknown None 107G 0s ago 2022-03-08T23:11:43.437 INFO:teuthology.orchestra.run.smithi077.stdout:smithi106 /dev/vg_nvme/lv_3 unknown None 107G 0s ago 2022-03-08T23:11:43.437 INFO:teuthology.orchestra.run.smithi077.stdout:smithi106 /dev/vg_nvme/lv_4 unknown None 107G 0s ago 2022-03-08T23:11:43.438 INFO:teuthology.orchestra.run.smithi077.stdout:smithi135 /dev/vg_nvme/lv_1 unknown None 107G 0s ago 2022-03-08T23:11:43.438 INFO:teuthology.orchestra.run.smithi077.stdout:smithi135 /dev/vg_nvme/lv_2 unknown None 107G 0s ago 2022-03-08T23:11:43.438 INFO:teuthology.orchestra.run.smithi077.stdout:smithi135 /dev/vg_nvme/lv_3 unknown None 107G 0s ago 2022-03-08T23:11:43.438 INFO:teuthology.orchestra.run.smithi077.stdout:smithi135 /dev/vg_nvme/lv_4 unknown None 107G 0s ago 2022-03-08T23:11:43.459 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph orch apply rgw foo 2022-03-08T23:11:44.043 INFO:teuthology.orchestra.run.smithi077.stdout:Created 2022-03-08T23:11:44.044 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:11:44.044361 I | ceph-spec: adding finalizer "cephobjectstore.ceph.rook.io" on "foo" 2022-03-08T23:11:44.090 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph orch apply mds foo 2022-03-08T23:11:44.110 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:11:44.110063 I | op-mon: parsing mon endpoints: a=10.110.141.123:6789,b=10.98.41.230:6789,c=10.109.47.142:6789 2022-03-08T23:11:44.111 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:11:44.110131 I | ceph-spec: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3... 2022-03-08T23:11:44.733 INFO:teuthology.orchestra.run.smithi077.stdout:Created 2022-03-08T23:11:44.756 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph orch apply rbd-mirror 2022-03-08T23:11:44.811 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:11:44.738465 I | ceph-spec: adding finalizer "cephfilesystem.ceph.rook.io" on "foo" 2022-03-08T23:11:44.812 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:11:44.781163 W | ceph-file-controller: failed to set filesystem "foo" status to "". failed to update object "rook-ceph/foo" status: Operation cannot be fulfilled on cephfilesystems.ceph.rook.io "foo": the object has been modified; please apply your changes to the latest version and try again 2022-03-08T23:11:44.812 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:11:44.785462 I | op-mon: parsing mon endpoints: a=10.110.141.123:6789,b=10.98.41.230:6789,c=10.109.47.142:6789 2022-03-08T23:11:44.812 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:11:44.785496 I | ceph-spec: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3... 2022-03-08T23:11:45.356 INFO:teuthology.orchestra.run.smithi077.stdout:Success 2022-03-08T23:11:45.375 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph orch apply nfs foo --port 12777 2022-03-08T23:11:45.389 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:11:45.389368 I | op-mon: parsing mon endpoints: a=10.110.141.123:6789,b=10.98.41.230:6789,c=10.109.47.142:6789 2022-03-08T23:11:45.390 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:11:45.389464 I | ceph-spec: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3... 2022-03-08T23:11:49.504 INFO:teuthology.orchestra.run.smithi077.stdout:Unable to create NFS daemon, check logs for more traceback 2022-03-08T23:11:49.504 INFO:teuthology.orchestra.run.smithi077.stdout:'RookOrchestrator' object has no attribute 'open_ioctx' 2022-03-08T23:11:49.526 DEBUG:teuthology.run_tasks:Unwinding manager rook 2022-03-08T23:11:49.538 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph orch ls -f json 2022-03-08T23:11:49.861 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:11:49.860908 I | ceph-spec: detected ceph image version: "17.0.0-11025 quincy" 2022-03-08T23:11:50.332 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-03-08T23:11:50.333 INFO:teuthology.orchestra.run.smithi077.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:16.000000Z", "last_refresh": "2022-03-08T23:11:50.172311Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "mds.foo", "service_type": "mds", "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:11:50.172311Z", "running": 0, "size": 2}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:04.000000Z", "last_refresh": "2022-03-08T23:11:50.172311Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:24.000000Z", "last_refresh": "2022-03-08T23:11:50.172311Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:07:27.000000Z", "last_refresh": "2022-03-08T23:11:50.172311Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:11:50.172311Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_name": "rbd-mirror", "service_type": "rbd-mirror", "status": {"last_refresh": "2022-03-08T23:11:50.172311Z", "running": 0, "size": 1}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:11:50.172311Z", "running": 0, "size": 1}}] 2022-03-08T23:11:50.351 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph orch rm mds.foo 2022-03-08T23:11:51.051 INFO:teuthology.orchestra.run.smithi077.stdout:Removed cephfilesystems/foo 2022-03-08T23:11:51.074 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph orch rm rbd-mirror 2022-03-08T23:11:51.552 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:11:51.421342 I | ceph-object-controller: reconciling object store deployments 2022-03-08T23:11:51.552 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:11:51.463689 I | ceph-object-controller: ceph object store gateway service running at 10.111.128.69 2022-03-08T23:11:51.553 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:11:51.463721 I | ceph-object-controller: reconciling object store pools 2022-03-08T23:11:51.638 INFO:teuthology.orchestra.run.smithi077.stdout:Removed cephrbdmirrors/default-rbd-mirror 2022-03-08T23:11:51.659 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph orch rm rgw.foo 2022-03-08T23:11:51.946 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:11:51.945749 W | ceph-object-controller: failed to adjust the PG count for rgw metadata pools. using the general default. failed to get config setting "rgw_rados_pool_pg_num_min" for user "mon.": exit status 2 2022-03-08T23:11:52.195 INFO:teuthology.orchestra.run.smithi077.stdout:Removed cephobjectstores/foo 2022-03-08T23:11:52.216 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph orch ls -f json 2022-03-08T23:11:52.871 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:11:52.871252 I | ceph-spec: detected ceph image version: "17.0.0-11025 quincy" 2022-03-08T23:11:53.037 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-03-08T23:11:53.037 INFO:teuthology.orchestra.run.smithi077.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:16.000000Z", "last_refresh": "2022-03-08T23:11:52.809912Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "mds.foo", "service_type": "mds", "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:11:52.809912Z", "running": 0, "size": 2}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:04.000000Z", "last_refresh": "2022-03-08T23:11:52.809912Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:24.000000Z", "last_refresh": "2022-03-08T23:11:52.809912Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:07:27.000000Z", "last_refresh": "2022-03-08T23:11:52.809912Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:11:52.809912Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:11:52.809912Z", "running": 0, "size": 1}}] 2022-03-08T23:11:54.099 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:11:54.098858 I | ceph-file-controller: start running mdses for filesystem "foo" 2022-03-08T23:11:54.400 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:11:54.399571 I | ceph-spec: detected ceph image version: "17.0.0-11025 quincy" 2022-03-08T23:11:54.775 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:11:54.775262 I | cephclient: getting or creating ceph auth key "mds.foo-a" 2022-03-08T23:11:55.173 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:11:55.172608 W | ceph-rbd-mirror-controller: (DEPRECATED) use of peer secret names in CephRBDMirror is deprecated. Please use CephBlockPool CR to configure peer secret names and import peers. 2022-03-08T23:11:55.174 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:11:55.172643 I | ceph-rbd-mirror-controller: configure rbd-mirroring with 1 workers 2022-03-08T23:11:55.174 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:11:55.172655 I | cephclient: getting or creating ceph auth key "client.rbd-mirror.a" 2022-03-08T23:11:57.807 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:11:57.806805 I | op-mds: setting mds config flags 2022-03-08T23:11:57.807 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:11:57.806844 I | op-config: setting "mds.foo-a"="mds_join_fs"="foo" option to the mon configuration database 2022-03-08T23:11:57.967 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:11:57.967085 I | ceph-rbd-mirror-controller: "rook-ceph-rbd-mirror-a" deployment started 2022-03-08T23:11:58.239 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:11:58.238417 I | ceph-spec: object "rook-ceph-rbd-mirror-a" matched on delete, reconciling 2022-03-08T23:11:58.293 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:11:58.292446 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi106": the object has been modified; please apply your changes to the latest version and try again 2022-03-08T23:11:58.337 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:11:58.337549 I | op-config: successfully set "mds.foo-a"="mds_join_fs"="foo" option to the mon configuration database 2022-03-08T23:11:58.420 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:11:58.416713 I | cephclient: getting or creating ceph auth key "mds.foo-b" 2022-03-08T23:11:59.561 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:11:59.560833 I | cephclient: reconciling replicated pool foo.rgw.buckets.index succeeded 2022-03-08T23:11:59.561 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:11:59.560863 I | cephclient: reconciling replicated pool foo.rgw.control succeeded 2022-03-08T23:11:59.564 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:11:59.561121 I | cephclient: reconciling replicated pool foo.rgw.meta succeeded 2022-03-08T23:12:00.296 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:12:00.295621 I | op-mds: setting mds config flags 2022-03-08T23:12:00.296 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:12:00.295658 I | op-config: setting "mds.foo-b"="mds_join_fs"="foo" option to the mon configuration database 2022-03-08T23:12:00.544 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:12:00.543906 I | cephclient: reconciling replicated pool foo.rgw.buckets.non-ec succeeded 2022-03-08T23:12:00.549 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:12:00.547360 I | cephclient: reconciling replicated pool foo.rgw.log succeeded 2022-03-08T23:12:01.277 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:12:01.276987 I | op-config: successfully set "mds.foo-b"="mds_join_fs"="foo" option to the mon configuration database 2022-03-08T23:12:01.990 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:12:01.989722 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool ".rgw.root": failed to tag pool ".rgw.root" for application "rook-ceph-rgw": failed to get application for pool ".rgw.root": failed to get current application for pool .rgw.root: exit status 2 2022-03-08T23:12:02.023 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:12:02.022555 I | op-mon: parsing mon endpoints: a=10.110.141.123:6789,b=10.98.41.230:6789,c=10.109.47.142:6789 2022-03-08T23:12:03.061 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph orch ls -f json 2022-03-08T23:12:03.333 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:12:03.332483 I | ceph-file-controller: creating filesystem "foo" 2022-03-08T23:12:03.739 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-03-08T23:12:03.739 INFO:teuthology.orchestra.run.smithi077.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:16.000000Z", "last_refresh": "2022-03-08T23:12:03.575754Z", "running": 5, "size": 3}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "mds.foo", "service_type": "mds", "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:12:03.575754Z", "running": 0, "size": 2}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:04.000000Z", "last_refresh": "2022-03-08T23:12:03.575754Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:24.000000Z", "last_refresh": "2022-03-08T23:12:03.575754Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:07:27.000000Z", "last_refresh": "2022-03-08T23:12:03.575754Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:12:03.575754Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:12:03.575754Z", "running": 0, "size": 1}}] 2022-03-08T23:12:06.739 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:12:06.738331 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T23:12:06.727+0000 7ffb60281380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T23:12:06.739 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08T23:12:06.727+0000 7ffb60281380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T23:12:06.739 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08T23:12:06.727+0000 7ffb60281380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T23:12:06.740 INFO:tasks.rook.operator.smithi077.stdout:couldn't init storage provider. : exit status 5 2022-03-08T23:12:06.772 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:12:06.772246 I | op-mon: parsing mon endpoints: a=10.110.141.123:6789,b=10.98.41.230:6789,c=10.109.47.142:6789 2022-03-08T23:12:07.330 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:12:07.329459 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T23:12:07.319+0000 7fd9c1f8f380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T23:12:07.330 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08T23:12:07.319+0000 7fd9c1f8f380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T23:12:07.330 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08T23:12:07.319+0000 7fd9c1f8f380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T23:12:07.331 INFO:tasks.rook.operator.smithi077.stdout:couldn't init storage provider. : exit status 5 2022-03-08T23:12:07.337 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:12:07.337613 I | op-mon: parsing mon endpoints: a=10.110.141.123:6789,b=10.98.41.230:6789,c=10.109.47.142:6789 2022-03-08T23:12:08.339 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:12:08.338325 I | cephclient: reconciling replicated pool foo-metadata succeeded 2022-03-08T23:12:08.442 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:12:08.441970 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T23:12:08.431+0000 7fa6cedf2380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T23:12:08.443 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08T23:12:08.431+0000 7fa6cedf2380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T23:12:08.443 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08T23:12:08.431+0000 7fa6cedf2380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T23:12:08.443 INFO:tasks.rook.operator.smithi077.stdout:couldn't init storage provider. : exit status 5 2022-03-08T23:12:08.454 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:12:08.454109 I | op-mon: parsing mon endpoints: a=10.110.141.123:6789,b=10.98.41.230:6789,c=10.109.47.142:6789 2022-03-08T23:12:09.296 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:12:09.295253 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T23:12:09.283+0000 7fd58bf15380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T23:12:09.296 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08T23:12:09.283+0000 7fd58bf15380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T23:12:09.296 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08T23:12:09.283+0000 7fd58bf15380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T23:12:09.297 INFO:tasks.rook.operator.smithi077.stdout:couldn't init storage provider. : exit status 5 2022-03-08T23:12:09.307 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:12:09.306546 I | op-mon: parsing mon endpoints: a=10.110.141.123:6789,b=10.98.41.230:6789,c=10.109.47.142:6789 2022-03-08T23:12:10.313 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:12:10.313065 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T23:12:10.299+0000 7f87da29d380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T23:12:10.314 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08T23:12:10.299+0000 7f87da29d380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T23:12:10.314 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08T23:12:10.299+0000 7f87da29d380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T23:12:10.314 INFO:tasks.rook.operator.smithi077.stdout:couldn't init storage provider. : exit status 5 2022-03-08T23:12:10.464 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:12:10.464042 I | op-mon: parsing mon endpoints: a=10.110.141.123:6789,b=10.98.41.230:6789,c=10.109.47.142:6789 2022-03-08T23:12:11.327 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:12:11.326663 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T23:12:11.315+0000 7fef29133380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T23:12:11.327 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08T23:12:11.315+0000 7fef29133380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T23:12:11.328 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08T23:12:11.315+0000 7fef29133380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T23:12:11.328 INFO:tasks.rook.operator.smithi077.stdout:couldn't init storage provider. : exit status 5 2022-03-08T23:12:11.332 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:12:11.332038 I | op-mon: parsing mon endpoints: a=10.110.141.123:6789,b=10.98.41.230:6789,c=10.109.47.142:6789 2022-03-08T23:12:12.016 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:12:12.015698 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T23:12:12.003+0000 7fc8a4f9d380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T23:12:12.017 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08T23:12:12.003+0000 7fc8a4f9d380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T23:12:12.017 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08T23:12:12.003+0000 7fc8a4f9d380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T23:12:12.018 INFO:tasks.rook.operator.smithi077.stdout:couldn't init storage provider. : exit status 5 2022-03-08T23:12:12.053 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:12:12.052815 I | op-mon: parsing mon endpoints: a=10.110.141.123:6789,b=10.98.41.230:6789,c=10.109.47.142:6789 2022-03-08T23:12:12.647 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:12:12.646401 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T23:12:12.635+0000 7fccad355380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T23:12:12.647 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08T23:12:12.635+0000 7fccad355380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T23:12:12.648 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08T23:12:12.635+0000 7fccad355380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T23:12:12.648 INFO:tasks.rook.operator.smithi077.stdout:couldn't init storage provider. : exit status 5 2022-03-08T23:12:12.661 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:12:12.661228 I | op-mon: parsing mon endpoints: a=10.110.141.123:6789,b=10.98.41.230:6789,c=10.109.47.142:6789 2022-03-08T23:12:13.101 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:12:13.101152 I | cephclient: reconciling replicated pool foo-data0 succeeded 2022-03-08T23:12:13.409 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:12:13.408706 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T23:12:13.399+0000 7f3d88c56380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T23:12:13.410 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08T23:12:13.399+0000 7f3d88c56380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T23:12:13.410 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08T23:12:13.399+0000 7f3d88c56380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T23:12:13.411 INFO:tasks.rook.operator.smithi077.stdout:couldn't init storage provider. : exit status 5 2022-03-08T23:12:13.763 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph orch ls -f json 2022-03-08T23:12:13.983 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:12:13.982500 I | cephclient: creating filesystem "foo" with metadata pool "foo-metadata" and data pools [foo-data0] 2022-03-08T23:12:14.529 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-03-08T23:12:14.530 INFO:teuthology.orchestra.run.smithi077.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:16.000000Z", "last_refresh": "2022-03-08T23:12:14.350422Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "mds.foo", "service_type": "mds", "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:12:14.350422Z", "running": 0, "size": 2}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:04.000000Z", "last_refresh": "2022-03-08T23:12:14.350422Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:24.000000Z", "last_refresh": "2022-03-08T23:12:14.350422Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:07:27.000000Z", "last_refresh": "2022-03-08T23:12:14.350422Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:12:14.350422Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:12:14.350422Z", "running": 0, "size": 1}}] 2022-03-08T23:12:15.795 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:12:15.794429 I | ceph-file-controller: created filesystem "foo" on 1 data pool(s) and metadata pool "foo-metadata" 2022-03-08T23:12:15.795 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:12:15.794460 I | cephclient: setting allow_standby_replay for filesystem "foo" 2022-03-08T23:12:15.975 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:12:15.974614 I | op-mon: parsing mon endpoints: a=10.110.141.123:6789,b=10.98.41.230:6789,c=10.109.47.142:6789 2022-03-08T23:12:16.587 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:12:16.586653 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T23:12:16.575+0000 7f9ca85ef380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T23:12:16.587 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08T23:12:16.575+0000 7f9ca85ef380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T23:12:16.588 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08T23:12:16.575+0000 7f9ca85ef380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T23:12:16.588 INFO:tasks.rook.operator.smithi077.stdout:couldn't init storage provider. : exit status 5 2022-03-08T23:12:16.899 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:12:16.898965 W | ceph-file-controller: failed to set filesystem "foo" status to "". failed to update object "rook-ceph/foo" status: Operation cannot be fulfilled on cephfilesystems.ceph.rook.io "foo": the object has been modified; please apply your changes to the latest version and try again 2022-03-08T23:12:16.904 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:12:16.904706 I | op-mon: parsing mon endpoints: a=10.110.141.123:6789,b=10.98.41.230:6789,c=10.109.47.142:6789 2022-03-08T23:12:16.908 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:12:16.906646 I | ceph-file-controller: CephFilesystem "rook-ceph/foo" can be deleted safely. deleting CephFilesystem "rook-ceph/foo" 2022-03-08T23:12:17.707 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:12:17.707056 I | op-config: deleting "mds_join_fs" option from the mon configuration database 2022-03-08T23:12:18.160 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:12:18.159547 I | op-config: successfully deleted "mds_join_fs" option from the mon configuration database 2022-03-08T23:12:18.160 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:12:18.159583 I | op-mds: successfully deleted mds config for "mds.foo-a" in mon configuration database 2022-03-08T23:12:18.161 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:12:18.159593 I | cephclient: deleting ceph auth "mds.foo-a" 2022-03-08T23:12:18.805 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:12:18.804625 I | op-mds: successfully deleted mds CephX key for "mds.foo-a" 2022-03-08T23:12:19.135 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:12:19.135565 I | op-config: deleting "mds_join_fs" option from the mon configuration database 2022-03-08T23:12:19.590 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:12:19.589823 I | op-config: successfully deleted "mds_join_fs" option from the mon configuration database 2022-03-08T23:12:19.590 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:12:19.589855 I | op-mds: successfully deleted mds config for "mds.foo-b" in mon configuration database 2022-03-08T23:12:19.591 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:12:19.589864 I | cephclient: deleting ceph auth "mds.foo-b" 2022-03-08T23:12:20.212 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:12:20.211878 I | op-mds: successfully deleted mds CephX key for "mds.foo-b" 2022-03-08T23:12:20.213 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:12:20.211918 I | ceph-file-controller: downing filesystem "foo" 2022-03-08T23:12:20.924 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:12:20.924098 I | ceph-file-controller: downed filesystem "foo" 2022-03-08T23:12:21.729 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:12:21.728487 I | op-mon: parsing mon endpoints: a=10.110.141.123:6789,b=10.98.41.230:6789,c=10.109.47.142:6789 2022-03-08T23:12:22.404 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:12:22.404032 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T23:12:22.399+0000 7f1747a48380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T23:12:22.405 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08T23:12:22.399+0000 7f1747a48380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T23:12:22.405 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08T23:12:22.399+0000 7f1747a48380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T23:12:22.405 INFO:tasks.rook.operator.smithi077.stdout:couldn't init storage provider. : exit status 5 2022-03-08T23:12:23.031 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:12:23.030891 I | cephclient: no images/snapshosts present in pool "foo-metadata" 2022-03-08T23:12:23.031 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:12:23.030918 I | cephclient: purging pool "foo-metadata" (id=9) 2022-03-08T23:12:24.561 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph orch ls -f json 2022-03-08T23:12:25.260 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-03-08T23:12:25.261 INFO:teuthology.orchestra.run.smithi077.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:16.000000Z", "last_refresh": "2022-03-08T23:12:25.083793Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "mds.foo", "service_type": "mds", "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:12:25.083793Z", "running": 0, "size": 2}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:04.000000Z", "last_refresh": "2022-03-08T23:12:25.083793Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:24.000000Z", "last_refresh": "2022-03-08T23:12:25.083793Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:07:27.000000Z", "last_refresh": "2022-03-08T23:12:25.083793Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:12:25.083793Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:12:25.083793Z", "running": 0, "size": 1}}] 2022-03-08T23:12:26.161 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:12:26.160865 I | cephclient: purge completed for pool "foo-metadata" 2022-03-08T23:12:26.553 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:12:26.552642 I | cephclient: no images/snapshosts present in pool "foo-data0" 2022-03-08T23:12:26.553 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:12:26.552673 I | cephclient: purging pool "foo-data0" (id=10) 2022-03-08T23:12:29.449 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:12:29.448464 I | cephclient: purge completed for pool "foo-data0" 2022-03-08T23:12:29.449 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:12:29.448565 I | ceph-spec: removing finalizer "cephfilesystem.ceph.rook.io" on "foo" 2022-03-08T23:12:29.489 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:12:29.488848 I | ceph-spec: object "rook-ceph-mds-foo-b-keyring" matched on delete, reconciling 2022-03-08T23:12:29.544 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:12:29.543603 I | ceph-spec: object "rook-ceph-mds-foo-a-keyring" matched on delete, reconciling 2022-03-08T23:12:29.548 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:12:29.544480 I | ceph-spec: object "rook-ceph-mds-foo-b" matched on delete, reconciling 2022-03-08T23:12:29.549 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:12:29.544625 I | ceph-spec: object "rook-ceph-mds-foo-a" matched on delete, reconciling 2022-03-08T23:12:31.888 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:12:31.888042 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi135": the object has been modified; please apply your changes to the latest version and try again 2022-03-08T23:12:31.971 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:12:31.971109 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi135": the object has been modified; please apply your changes to the latest version and try again 2022-03-08T23:12:32.692 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:12:32.691413 I | op-mon: parsing mon endpoints: a=10.110.141.123:6789,b=10.98.41.230:6789,c=10.109.47.142:6789 2022-03-08T23:12:33.328 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:12:33.327385 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T23:12:33.319+0000 7f028d606380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T23:12:33.328 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08T23:12:33.319+0000 7f028d606380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T23:12:33.328 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08T23:12:33.319+0000 7f028d606380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T23:12:33.329 INFO:tasks.rook.operator.smithi077.stdout:couldn't init storage provider. : exit status 5 2022-03-08T23:12:35.284 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph orch ls -f json 2022-03-08T23:12:35.978 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-03-08T23:12:35.978 INFO:teuthology.orchestra.run.smithi077.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:16.000000Z", "last_refresh": "2022-03-08T23:12:35.796514Z", "running": 4, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:04.000000Z", "last_refresh": "2022-03-08T23:12:35.796514Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:24.000000Z", "last_refresh": "2022-03-08T23:12:35.796514Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:07:27.000000Z", "last_refresh": "2022-03-08T23:12:35.796514Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:12:35.796514Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:12:35.796514Z", "running": 0, "size": 1}}] 2022-03-08T23:12:46.001 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph orch ls -f json 2022-03-08T23:12:46.692 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-03-08T23:12:46.692 INFO:teuthology.orchestra.run.smithi077.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:16.000000Z", "last_refresh": "2022-03-08T23:12:46.497072Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:04.000000Z", "last_refresh": "2022-03-08T23:12:46.497072Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:24.000000Z", "last_refresh": "2022-03-08T23:12:46.497072Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:07:27.000000Z", "last_refresh": "2022-03-08T23:12:46.497072Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:12:46.497072Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:12:46.497072Z", "running": 0, "size": 1}}] 2022-03-08T23:12:53.812 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:12:53.812276 I | op-mon: parsing mon endpoints: a=10.110.141.123:6789,b=10.98.41.230:6789,c=10.109.47.142:6789 2022-03-08T23:12:54.584 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:12:54.584101 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T23:12:54.574+0000 7f3a06d8f380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T23:12:54.585 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08T23:12:54.574+0000 7f3a06d8f380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T23:12:54.585 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08T23:12:54.574+0000 7f3a06d8f380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T23:12:54.585 INFO:tasks.rook.operator.smithi077.stdout:couldn't init storage provider. : exit status 5 2022-03-08T23:12:56.716 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph orch ls -f json 2022-03-08T23:12:57.376 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-03-08T23:12:57.377 INFO:teuthology.orchestra.run.smithi077.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:16.000000Z", "last_refresh": "2022-03-08T23:12:57.219835Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:04.000000Z", "last_refresh": "2022-03-08T23:12:57.219835Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:24.000000Z", "last_refresh": "2022-03-08T23:12:57.219835Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:07:27.000000Z", "last_refresh": "2022-03-08T23:12:57.219835Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:12:57.219835Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:12:57.219835Z", "running": 0, "size": 1}}] 2022-03-08T23:13:07.399 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph orch ls -f json 2022-03-08T23:13:08.319 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-03-08T23:13:08.320 INFO:teuthology.orchestra.run.smithi077.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:16.000000Z", "last_refresh": "2022-03-08T23:13:07.968697Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:04.000000Z", "last_refresh": "2022-03-08T23:13:07.968697Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:24.000000Z", "last_refresh": "2022-03-08T23:13:07.968697Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:07:27.000000Z", "last_refresh": "2022-03-08T23:13:07.968697Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:13:07.968697Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:13:07.968697Z", "running": 0, "size": 1}}] 2022-03-08T23:13:18.339 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph orch ls -f json 2022-03-08T23:13:19.017 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-03-08T23:13:19.018 INFO:teuthology.orchestra.run.smithi077.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:16.000000Z", "last_refresh": "2022-03-08T23:13:18.857219Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:04.000000Z", "last_refresh": "2022-03-08T23:13:18.857219Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:24.000000Z", "last_refresh": "2022-03-08T23:13:18.857219Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:07:27.000000Z", "last_refresh": "2022-03-08T23:13:18.857219Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:13:18.857219Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:13:18.857219Z", "running": 0, "size": 1}}] 2022-03-08T23:13:29.042 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph orch ls -f json 2022-03-08T23:13:29.788 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-03-08T23:13:29.789 INFO:teuthology.orchestra.run.smithi077.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:16.000000Z", "last_refresh": "2022-03-08T23:13:29.594759Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:04.000000Z", "last_refresh": "2022-03-08T23:13:29.594759Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:24.000000Z", "last_refresh": "2022-03-08T23:13:29.594759Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:07:27.000000Z", "last_refresh": "2022-03-08T23:13:29.594759Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:13:29.594759Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:13:29.594759Z", "running": 0, "size": 1}}] 2022-03-08T23:13:35.554 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:13:35.553719 I | op-mon: parsing mon endpoints: a=10.110.141.123:6789,b=10.98.41.230:6789,c=10.109.47.142:6789 2022-03-08T23:13:36.209 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:13:36.208806 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T23:13:36.197+0000 7f8e8e1bb380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T23:13:36.209 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08T23:13:36.197+0000 7f8e8e1bb380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T23:13:36.209 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08T23:13:36.197+0000 7f8e8e1bb380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T23:13:36.210 INFO:tasks.rook.operator.smithi077.stdout:couldn't init storage provider. : exit status 5 2022-03-08T23:13:39.817 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph orch ls -f json 2022-03-08T23:13:40.521 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-03-08T23:13:40.521 INFO:teuthology.orchestra.run.smithi077.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:16.000000Z", "last_refresh": "2022-03-08T23:13:40.350843Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:04.000000Z", "last_refresh": "2022-03-08T23:13:40.350843Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:24.000000Z", "last_refresh": "2022-03-08T23:13:40.350843Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:07:27.000000Z", "last_refresh": "2022-03-08T23:13:40.350843Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:13:40.350843Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:13:40.350843Z", "running": 0, "size": 1}}] 2022-03-08T23:13:50.540 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph orch ls -f json 2022-03-08T23:13:51.255 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-03-08T23:13:51.255 INFO:teuthology.orchestra.run.smithi077.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:16.000000Z", "last_refresh": "2022-03-08T23:13:51.077913Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:04.000000Z", "last_refresh": "2022-03-08T23:13:51.077913Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:24.000000Z", "last_refresh": "2022-03-08T23:13:51.077913Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:07:27.000000Z", "last_refresh": "2022-03-08T23:13:51.077913Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:13:51.077913Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:13:51.077913Z", "running": 0, "size": 1}}] 2022-03-08T23:14:01.294 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph orch ls -f json 2022-03-08T23:14:01.959 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-03-08T23:14:01.959 INFO:teuthology.orchestra.run.smithi077.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:16.000000Z", "last_refresh": "2022-03-08T23:14:01.783036Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:04.000000Z", "last_refresh": "2022-03-08T23:14:01.783036Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:24.000000Z", "last_refresh": "2022-03-08T23:14:01.783036Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:07:27.000000Z", "last_refresh": "2022-03-08T23:14:01.783036Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:14:01.783036Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:14:01.783036Z", "running": 0, "size": 1}}] 2022-03-08T23:14:11.984 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph orch ls -f json 2022-03-08T23:14:12.660 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-03-08T23:14:12.660 INFO:teuthology.orchestra.run.smithi077.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:16.000000Z", "last_refresh": "2022-03-08T23:14:12.484458Z", "running": 4, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:04.000000Z", "last_refresh": "2022-03-08T23:14:12.484458Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:24.000000Z", "last_refresh": "2022-03-08T23:14:12.484458Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:07:27.000000Z", "last_refresh": "2022-03-08T23:14:12.484458Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:14:12.484458Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:14:12.484458Z", "running": 0, "size": 1}}] 2022-03-08T23:14:22.683 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph orch ls -f json 2022-03-08T23:14:23.394 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-03-08T23:14:23.395 INFO:teuthology.orchestra.run.smithi077.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:16.000000Z", "last_refresh": "2022-03-08T23:14:23.231926Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:04.000000Z", "last_refresh": "2022-03-08T23:14:23.231926Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:24.000000Z", "last_refresh": "2022-03-08T23:14:23.231926Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:07:27.000000Z", "last_refresh": "2022-03-08T23:14:23.231926Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:14:23.231926Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:14:23.231926Z", "running": 0, "size": 1}}] 2022-03-08T23:14:33.415 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph orch ls -f json 2022-03-08T23:14:34.113 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-03-08T23:14:34.113 INFO:teuthology.orchestra.run.smithi077.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:16.000000Z", "last_refresh": "2022-03-08T23:14:33.935219Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:04.000000Z", "last_refresh": "2022-03-08T23:14:33.935219Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:24.000000Z", "last_refresh": "2022-03-08T23:14:33.935219Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:07:27.000000Z", "last_refresh": "2022-03-08T23:14:33.935219Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:14:33.935219Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:14:33.935219Z", "running": 0, "size": 1}}] 2022-03-08T23:14:44.137 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph orch ls -f json 2022-03-08T23:14:44.836 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-03-08T23:14:44.836 INFO:teuthology.orchestra.run.smithi077.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:16.000000Z", "last_refresh": "2022-03-08T23:14:44.651557Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:04.000000Z", "last_refresh": "2022-03-08T23:14:44.651557Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:24.000000Z", "last_refresh": "2022-03-08T23:14:44.651557Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:07:27.000000Z", "last_refresh": "2022-03-08T23:14:44.651557Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:14:44.651557Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:14:44.651557Z", "running": 0, "size": 1}}] 2022-03-08T23:14:54.859 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph orch ls -f json 2022-03-08T23:14:55.776 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-03-08T23:14:55.777 INFO:teuthology.orchestra.run.smithi077.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:16.000000Z", "last_refresh": "2022-03-08T23:14:55.364509Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:04.000000Z", "last_refresh": "2022-03-08T23:14:55.364509Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:24.000000Z", "last_refresh": "2022-03-08T23:14:55.364509Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:07:27.000000Z", "last_refresh": "2022-03-08T23:14:55.364509Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:14:55.364509Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:14:55.364509Z", "running": 0, "size": 1}}] 2022-03-08T23:14:58.137 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:14:58.137266 I | op-mon: parsing mon endpoints: a=10.110.141.123:6789,b=10.98.41.230:6789,c=10.109.47.142:6789 2022-03-08T23:14:58.798 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:14:58.797457 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T23:14:58.787+0000 7f3e35ab3380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T23:14:58.798 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08T23:14:58.787+0000 7f3e35ab3380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T23:14:58.798 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08T23:14:58.787+0000 7f3e35ab3380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T23:14:58.798 INFO:tasks.rook.operator.smithi077.stdout:couldn't init storage provider. : exit status 5 2022-03-08T23:15:05.805 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph orch ls -f json 2022-03-08T23:15:06.595 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-03-08T23:15:06.596 INFO:teuthology.orchestra.run.smithi077.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:16.000000Z", "last_refresh": "2022-03-08T23:15:06.353000Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:04.000000Z", "last_refresh": "2022-03-08T23:15:06.353000Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:24.000000Z", "last_refresh": "2022-03-08T23:15:06.353000Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:07:27.000000Z", "last_refresh": "2022-03-08T23:15:06.353000Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:15:06.353000Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:15:06.353000Z", "running": 0, "size": 1}}] 2022-03-08T23:15:16.616 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph orch ls -f json 2022-03-08T23:15:17.297 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-03-08T23:15:17.297 INFO:teuthology.orchestra.run.smithi077.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:16.000000Z", "last_refresh": "2022-03-08T23:15:17.122285Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:04.000000Z", "last_refresh": "2022-03-08T23:15:17.122285Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:24.000000Z", "last_refresh": "2022-03-08T23:15:17.122285Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:07:27.000000Z", "last_refresh": "2022-03-08T23:15:17.122285Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:15:17.122285Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:15:17.122285Z", "running": 0, "size": 1}}] 2022-03-08T23:15:27.321 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph orch ls -f json 2022-03-08T23:15:28.046 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-03-08T23:15:28.046 INFO:teuthology.orchestra.run.smithi077.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:16.000000Z", "last_refresh": "2022-03-08T23:15:27.841094Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:04.000000Z", "last_refresh": "2022-03-08T23:15:27.841094Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:24.000000Z", "last_refresh": "2022-03-08T23:15:27.841094Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:07:27.000000Z", "last_refresh": "2022-03-08T23:15:27.841094Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:15:27.841094Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:15:27.841094Z", "running": 0, "size": 1}}] 2022-03-08T23:15:38.069 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph orch ls -f json 2022-03-08T23:15:38.805 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-03-08T23:15:38.806 INFO:teuthology.orchestra.run.smithi077.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:16.000000Z", "last_refresh": "2022-03-08T23:15:38.617068Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:04.000000Z", "last_refresh": "2022-03-08T23:15:38.617068Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:24.000000Z", "last_refresh": "2022-03-08T23:15:38.617068Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:07:27.000000Z", "last_refresh": "2022-03-08T23:15:38.617068Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:15:38.617068Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:15:38.617068Z", "running": 0, "size": 1}}] 2022-03-08T23:15:48.824 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph orch ls -f json 2022-03-08T23:15:49.535 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-03-08T23:15:49.535 INFO:teuthology.orchestra.run.smithi077.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:16.000000Z", "last_refresh": "2022-03-08T23:15:49.368524Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:04.000000Z", "last_refresh": "2022-03-08T23:15:49.368524Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:24.000000Z", "last_refresh": "2022-03-08T23:15:49.368524Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:07:27.000000Z", "last_refresh": "2022-03-08T23:15:49.368524Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:15:49.368524Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:15:49.368524Z", "running": 0, "size": 1}}] 2022-03-08T23:15:59.556 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph orch ls -f json 2022-03-08T23:16:00.265 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-03-08T23:16:00.266 INFO:teuthology.orchestra.run.smithi077.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:16.000000Z", "last_refresh": "2022-03-08T23:16:00.092707Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:04.000000Z", "last_refresh": "2022-03-08T23:16:00.092707Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:24.000000Z", "last_refresh": "2022-03-08T23:16:00.092707Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:07:27.000000Z", "last_refresh": "2022-03-08T23:16:00.092707Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:16:00.092707Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:16:00.092707Z", "running": 0, "size": 1}}] 2022-03-08T23:16:10.288 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph orch ls -f json 2022-03-08T23:16:11.028 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-03-08T23:16:11.029 INFO:teuthology.orchestra.run.smithi077.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:16.000000Z", "last_refresh": "2022-03-08T23:16:10.874895Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:04.000000Z", "last_refresh": "2022-03-08T23:16:10.874895Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:24.000000Z", "last_refresh": "2022-03-08T23:16:10.874895Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:07:27.000000Z", "last_refresh": "2022-03-08T23:16:10.874895Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:16:10.874895Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:16:10.874895Z", "running": 0, "size": 1}}] 2022-03-08T23:16:21.054 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph orch ls -f json 2022-03-08T23:16:21.790 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-03-08T23:16:21.791 INFO:teuthology.orchestra.run.smithi077.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:16.000000Z", "last_refresh": "2022-03-08T23:16:21.601960Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:04.000000Z", "last_refresh": "2022-03-08T23:16:21.601960Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:24.000000Z", "last_refresh": "2022-03-08T23:16:21.601960Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:07:27.000000Z", "last_refresh": "2022-03-08T23:16:21.601960Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:16:21.601960Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:16:21.601960Z", "running": 0, "size": 1}}] 2022-03-08T23:16:31.814 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph orch ls -f json 2022-03-08T23:16:32.511 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-03-08T23:16:32.512 INFO:teuthology.orchestra.run.smithi077.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:16.000000Z", "last_refresh": "2022-03-08T23:16:32.356244Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:04.000000Z", "last_refresh": "2022-03-08T23:16:32.356244Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:24.000000Z", "last_refresh": "2022-03-08T23:16:32.356244Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:07:27.000000Z", "last_refresh": "2022-03-08T23:16:32.356244Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:16:32.356244Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:16:32.356244Z", "running": 0, "size": 1}}] 2022-03-08T23:16:42.534 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph orch ls -f json 2022-03-08T23:16:43.251 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-03-08T23:16:43.252 INFO:teuthology.orchestra.run.smithi077.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:16.000000Z", "last_refresh": "2022-03-08T23:16:43.052108Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:04.000000Z", "last_refresh": "2022-03-08T23:16:43.052108Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:24.000000Z", "last_refresh": "2022-03-08T23:16:43.052108Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:07:27.000000Z", "last_refresh": "2022-03-08T23:16:43.052108Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:16:43.052108Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:16:43.052108Z", "running": 0, "size": 1}}] 2022-03-08T23:16:53.274 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph orch ls -f json 2022-03-08T23:16:54.005 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-03-08T23:16:54.006 INFO:teuthology.orchestra.run.smithi077.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:16.000000Z", "last_refresh": "2022-03-08T23:16:53.806073Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:04.000000Z", "last_refresh": "2022-03-08T23:16:53.806073Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:24.000000Z", "last_refresh": "2022-03-08T23:16:53.806073Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:07:27.000000Z", "last_refresh": "2022-03-08T23:16:53.806073Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:16:53.806073Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:16:53.806073Z", "running": 0, "size": 1}}] 2022-03-08T23:17:04.052 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph orch ls -f json 2022-03-08T23:17:04.753 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-03-08T23:17:04.754 INFO:teuthology.orchestra.run.smithi077.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:16.000000Z", "last_refresh": "2022-03-08T23:17:04.553107Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:04.000000Z", "last_refresh": "2022-03-08T23:17:04.553107Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:24.000000Z", "last_refresh": "2022-03-08T23:17:04.553107Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:07:27.000000Z", "last_refresh": "2022-03-08T23:17:04.553107Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:17:04.553107Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:17:04.553107Z", "running": 0, "size": 1}}] 2022-03-08T23:17:14.773 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph orch ls -f json 2022-03-08T23:17:15.509 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-03-08T23:17:15.510 INFO:teuthology.orchestra.run.smithi077.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:16.000000Z", "last_refresh": "2022-03-08T23:17:15.297811Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:04.000000Z", "last_refresh": "2022-03-08T23:17:15.297811Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:24.000000Z", "last_refresh": "2022-03-08T23:17:15.297811Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:07:27.000000Z", "last_refresh": "2022-03-08T23:17:15.297811Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:17:15.297811Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:17:15.297811Z", "running": 0, "size": 1}}] 2022-03-08T23:17:25.533 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph orch ls -f json 2022-03-08T23:17:26.202 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-03-08T23:17:26.202 INFO:teuthology.orchestra.run.smithi077.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:16.000000Z", "last_refresh": "2022-03-08T23:17:26.037885Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:04.000000Z", "last_refresh": "2022-03-08T23:17:26.037885Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:24.000000Z", "last_refresh": "2022-03-08T23:17:26.037885Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:07:27.000000Z", "last_refresh": "2022-03-08T23:17:26.037885Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:17:26.037885Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:17:26.037885Z", "running": 0, "size": 1}}] 2022-03-08T23:17:36.227 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph orch ls -f json 2022-03-08T23:17:36.957 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-03-08T23:17:36.957 INFO:teuthology.orchestra.run.smithi077.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:16.000000Z", "last_refresh": "2022-03-08T23:17:36.782314Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:04.000000Z", "last_refresh": "2022-03-08T23:17:36.782314Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:24.000000Z", "last_refresh": "2022-03-08T23:17:36.782314Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:07:27.000000Z", "last_refresh": "2022-03-08T23:17:36.782314Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:17:36.782314Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:17:36.782314Z", "running": 0, "size": 1}}] 2022-03-08T23:17:42.644 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:17:42.644839 I | op-mon: parsing mon endpoints: a=10.110.141.123:6789,b=10.98.41.230:6789,c=10.109.47.142:6789 2022-03-08T23:17:43.272 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:17:43.272817 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T23:17:43.260+0000 7f8dd0d48380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T23:17:43.273 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08T23:17:43.260+0000 7f8dd0d48380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T23:17:43.273 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08T23:17:43.260+0000 7f8dd0d48380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T23:17:43.273 INFO:tasks.rook.operator.smithi077.stdout:couldn't init storage provider. : exit status 5 2022-03-08T23:17:46.981 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph orch ls -f json 2022-03-08T23:17:47.672 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-03-08T23:17:47.673 INFO:teuthology.orchestra.run.smithi077.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:16.000000Z", "last_refresh": "2022-03-08T23:17:47.509851Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:04.000000Z", "last_refresh": "2022-03-08T23:17:47.509851Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:24.000000Z", "last_refresh": "2022-03-08T23:17:47.509851Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:07:27.000000Z", "last_refresh": "2022-03-08T23:17:47.509851Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:17:47.509851Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:17:47.509851Z", "running": 0, "size": 1}}] 2022-03-08T23:17:57.695 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph orch ls -f json 2022-03-08T23:17:58.397 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-03-08T23:17:58.397 INFO:teuthology.orchestra.run.smithi077.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:16.000000Z", "last_refresh": "2022-03-08T23:17:58.222422Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:04.000000Z", "last_refresh": "2022-03-08T23:17:58.222422Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:24.000000Z", "last_refresh": "2022-03-08T23:17:58.222422Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:07:27.000000Z", "last_refresh": "2022-03-08T23:17:58.222422Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:17:58.222422Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:17:58.222422Z", "running": 0, "size": 1}}] 2022-03-08T23:18:08.427 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph orch ls -f json 2022-03-08T23:18:09.088 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-03-08T23:18:09.088 INFO:teuthology.orchestra.run.smithi077.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:16.000000Z", "last_refresh": "2022-03-08T23:18:08.919500Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:04.000000Z", "last_refresh": "2022-03-08T23:18:08.919500Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:24.000000Z", "last_refresh": "2022-03-08T23:18:08.919500Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:07:27.000000Z", "last_refresh": "2022-03-08T23:18:08.919500Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:18:08.919500Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:18:08.919500Z", "running": 0, "size": 1}}] 2022-03-08T23:18:19.109 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph orch ls -f json 2022-03-08T23:18:19.852 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-03-08T23:18:19.852 INFO:teuthology.orchestra.run.smithi077.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:16.000000Z", "last_refresh": "2022-03-08T23:18:19.642975Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:04.000000Z", "last_refresh": "2022-03-08T23:18:19.642975Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:24.000000Z", "last_refresh": "2022-03-08T23:18:19.642975Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:07:27.000000Z", "last_refresh": "2022-03-08T23:18:19.642975Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:18:19.642975Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:18:19.642975Z", "running": 0, "size": 1}}] 2022-03-08T23:18:29.877 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph orch ls -f json 2022-03-08T23:18:30.621 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-03-08T23:18:30.621 INFO:teuthology.orchestra.run.smithi077.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:16.000000Z", "last_refresh": "2022-03-08T23:18:30.445704Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:04.000000Z", "last_refresh": "2022-03-08T23:18:30.445704Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:24.000000Z", "last_refresh": "2022-03-08T23:18:30.445704Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:07:27.000000Z", "last_refresh": "2022-03-08T23:18:30.445704Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:18:30.445704Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:18:30.445704Z", "running": 0, "size": 1}}] 2022-03-08T23:18:40.645 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph orch ls -f json 2022-03-08T23:18:41.358 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-03-08T23:18:41.359 INFO:teuthology.orchestra.run.smithi077.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:16.000000Z", "last_refresh": "2022-03-08T23:18:41.187539Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:04.000000Z", "last_refresh": "2022-03-08T23:18:41.187539Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:24.000000Z", "last_refresh": "2022-03-08T23:18:41.187539Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:07:27.000000Z", "last_refresh": "2022-03-08T23:18:41.187539Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:18:41.187539Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:18:41.187539Z", "running": 0, "size": 1}}] 2022-03-08T23:18:51.381 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph orch ls -f json 2022-03-08T23:18:52.063 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-03-08T23:18:52.064 INFO:teuthology.orchestra.run.smithi077.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:16.000000Z", "last_refresh": "2022-03-08T23:18:51.903485Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:04.000000Z", "last_refresh": "2022-03-08T23:18:51.903485Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:24.000000Z", "last_refresh": "2022-03-08T23:18:51.903485Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:07:27.000000Z", "last_refresh": "2022-03-08T23:18:51.903485Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:18:51.903485Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:18:51.903485Z", "running": 0, "size": 1}}] 2022-03-08T23:19:02.085 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph orch ls -f json 2022-03-08T23:19:02.789 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-03-08T23:19:02.789 INFO:teuthology.orchestra.run.smithi077.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:16.000000Z", "last_refresh": "2022-03-08T23:19:02.628529Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:04.000000Z", "last_refresh": "2022-03-08T23:19:02.628529Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:24.000000Z", "last_refresh": "2022-03-08T23:19:02.628529Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:07:27.000000Z", "last_refresh": "2022-03-08T23:19:02.628529Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:19:02.628529Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:19:02.628529Z", "running": 0, "size": 1}}] 2022-03-08T23:19:12.810 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph orch ls -f json 2022-03-08T23:19:13.537 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-03-08T23:19:13.538 INFO:teuthology.orchestra.run.smithi077.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:16.000000Z", "last_refresh": "2022-03-08T23:19:13.361157Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:04.000000Z", "last_refresh": "2022-03-08T23:19:13.361157Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:24.000000Z", "last_refresh": "2022-03-08T23:19:13.361157Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:07:27.000000Z", "last_refresh": "2022-03-08T23:19:13.361157Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:19:13.361157Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:19:13.361157Z", "running": 0, "size": 1}}] 2022-03-08T23:19:23.557 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph orch ls -f json 2022-03-08T23:19:24.244 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-03-08T23:19:24.244 INFO:teuthology.orchestra.run.smithi077.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:16.000000Z", "last_refresh": "2022-03-08T23:19:24.090657Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:04.000000Z", "last_refresh": "2022-03-08T23:19:24.090657Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:24.000000Z", "last_refresh": "2022-03-08T23:19:24.090657Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:07:27.000000Z", "last_refresh": "2022-03-08T23:19:24.090657Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:19:24.090657Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:19:24.090657Z", "running": 0, "size": 1}}] 2022-03-08T23:19:34.266 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph orch ls -f json 2022-03-08T23:19:35.026 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-03-08T23:19:35.027 INFO:teuthology.orchestra.run.smithi077.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:16.000000Z", "last_refresh": "2022-03-08T23:19:34.796848Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:04.000000Z", "last_refresh": "2022-03-08T23:19:34.796848Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:24.000000Z", "last_refresh": "2022-03-08T23:19:34.796848Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:07:27.000000Z", "last_refresh": "2022-03-08T23:19:34.796848Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:19:34.796848Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:19:34.796848Z", "running": 0, "size": 1}}] 2022-03-08T23:19:45.051 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph orch ls -f json 2022-03-08T23:19:45.761 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-03-08T23:19:45.762 INFO:teuthology.orchestra.run.smithi077.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:16.000000Z", "last_refresh": "2022-03-08T23:19:45.614797Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:04.000000Z", "last_refresh": "2022-03-08T23:19:45.614797Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:24.000000Z", "last_refresh": "2022-03-08T23:19:45.614797Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:07:27.000000Z", "last_refresh": "2022-03-08T23:19:45.614797Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:19:45.614797Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:19:45.614797Z", "running": 0, "size": 1}}] 2022-03-08T23:19:55.781 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph orch ls -f json 2022-03-08T23:19:56.474 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-03-08T23:19:56.474 INFO:teuthology.orchestra.run.smithi077.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:16.000000Z", "last_refresh": "2022-03-08T23:19:56.312852Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:04.000000Z", "last_refresh": "2022-03-08T23:19:56.312852Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:24.000000Z", "last_refresh": "2022-03-08T23:19:56.312852Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:07:27.000000Z", "last_refresh": "2022-03-08T23:19:56.312852Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:19:56.312852Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:19:56.312852Z", "running": 0, "size": 1}}] 2022-03-08T23:20:06.504 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph orch ls -f json 2022-03-08T23:20:07.168 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-03-08T23:20:07.168 INFO:teuthology.orchestra.run.smithi077.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:16.000000Z", "last_refresh": "2022-03-08T23:20:07.009339Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:04.000000Z", "last_refresh": "2022-03-08T23:20:07.009339Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:24.000000Z", "last_refresh": "2022-03-08T23:20:07.009339Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:07:27.000000Z", "last_refresh": "2022-03-08T23:20:07.009339Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:20:07.009339Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:20:07.009339Z", "running": 0, "size": 1}}] 2022-03-08T23:20:17.187 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph orch ls -f json 2022-03-08T23:20:17.911 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-03-08T23:20:17.912 INFO:teuthology.orchestra.run.smithi077.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:16.000000Z", "last_refresh": "2022-03-08T23:20:17.754721Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:04.000000Z", "last_refresh": "2022-03-08T23:20:17.754721Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:24.000000Z", "last_refresh": "2022-03-08T23:20:17.754721Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:07:27.000000Z", "last_refresh": "2022-03-08T23:20:17.754721Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:20:17.754721Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:20:17.754721Z", "running": 0, "size": 1}}] 2022-03-08T23:20:27.939 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph orch ls -f json 2022-03-08T23:20:28.668 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-03-08T23:20:28.668 INFO:teuthology.orchestra.run.smithi077.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:16.000000Z", "last_refresh": "2022-03-08T23:20:28.478716Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:04.000000Z", "last_refresh": "2022-03-08T23:20:28.478716Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:24.000000Z", "last_refresh": "2022-03-08T23:20:28.478716Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:07:27.000000Z", "last_refresh": "2022-03-08T23:20:28.478716Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:20:28.478716Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:20:28.478716Z", "running": 0, "size": 1}}] 2022-03-08T23:20:38.690 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph orch ls -f json 2022-03-08T23:20:39.385 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-03-08T23:20:39.386 INFO:teuthology.orchestra.run.smithi077.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:16.000000Z", "last_refresh": "2022-03-08T23:20:39.220141Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:04.000000Z", "last_refresh": "2022-03-08T23:20:39.220141Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:24.000000Z", "last_refresh": "2022-03-08T23:20:39.220141Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:07:27.000000Z", "last_refresh": "2022-03-08T23:20:39.220141Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:20:39.220141Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:20:39.220141Z", "running": 0, "size": 1}}] 2022-03-08T23:20:49.409 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph orch ls -f json 2022-03-08T23:20:50.088 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-03-08T23:20:50.088 INFO:teuthology.orchestra.run.smithi077.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:16.000000Z", "last_refresh": "2022-03-08T23:20:49.913412Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:04.000000Z", "last_refresh": "2022-03-08T23:20:49.913412Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:24.000000Z", "last_refresh": "2022-03-08T23:20:49.913412Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:07:27.000000Z", "last_refresh": "2022-03-08T23:20:49.913412Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:20:49.913412Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:20:49.913412Z", "running": 0, "size": 1}}] 2022-03-08T23:21:00.136 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph orch ls -f json 2022-03-08T23:21:00.868 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-03-08T23:21:00.868 INFO:teuthology.orchestra.run.smithi077.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:16.000000Z", "last_refresh": "2022-03-08T23:21:00.713862Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:04.000000Z", "last_refresh": "2022-03-08T23:21:00.713862Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:24.000000Z", "last_refresh": "2022-03-08T23:21:00.713862Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:07:27.000000Z", "last_refresh": "2022-03-08T23:21:00.713862Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:21:00.713862Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:21:00.713862Z", "running": 0, "size": 1}}] 2022-03-08T23:21:10.890 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph orch ls -f json 2022-03-08T23:21:11.591 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-03-08T23:21:11.592 INFO:teuthology.orchestra.run.smithi077.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:16.000000Z", "last_refresh": "2022-03-08T23:21:11.443708Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:04.000000Z", "last_refresh": "2022-03-08T23:21:11.443708Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:24.000000Z", "last_refresh": "2022-03-08T23:21:11.443708Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:07:27.000000Z", "last_refresh": "2022-03-08T23:21:11.443708Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:21:11.443708Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:21:11.443708Z", "running": 0, "size": 1}}] 2022-03-08T23:21:21.617 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph orch ls -f json 2022-03-08T23:21:22.271 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-03-08T23:21:22.271 INFO:teuthology.orchestra.run.smithi077.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:16.000000Z", "last_refresh": "2022-03-08T23:21:22.111095Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:04.000000Z", "last_refresh": "2022-03-08T23:21:22.111095Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:24.000000Z", "last_refresh": "2022-03-08T23:21:22.111095Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:07:27.000000Z", "last_refresh": "2022-03-08T23:21:22.111095Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:21:22.111095Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:21:22.111095Z", "running": 0, "size": 1}}] 2022-03-08T23:21:32.295 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph orch ls -f json 2022-03-08T23:21:32.973 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-03-08T23:21:32.974 INFO:teuthology.orchestra.run.smithi077.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:16.000000Z", "last_refresh": "2022-03-08T23:21:32.827588Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:04.000000Z", "last_refresh": "2022-03-08T23:21:32.827588Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:24.000000Z", "last_refresh": "2022-03-08T23:21:32.827588Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:07:27.000000Z", "last_refresh": "2022-03-08T23:21:32.827588Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:21:32.827588Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:21:32.827588Z", "running": 0, "size": 1}}] 2022-03-08T23:21:42.996 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph orch ls -f json 2022-03-08T23:21:46.286 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-03-08T23:21:46.287 INFO:teuthology.orchestra.run.smithi077.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:16.000000Z", "last_refresh": "2022-03-08T23:21:46.093811Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:04.000000Z", "last_refresh": "2022-03-08T23:21:46.093811Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:24.000000Z", "last_refresh": "2022-03-08T23:21:46.093811Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:07:27.000000Z", "last_refresh": "2022-03-08T23:21:46.093811Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:21:46.093811Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:21:46.093811Z", "running": 0, "size": 1}}] 2022-03-08T23:21:56.310 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph orch ls -f json 2022-03-08T23:21:57.024 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-03-08T23:21:57.024 INFO:teuthology.orchestra.run.smithi077.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:16.000000Z", "last_refresh": "2022-03-08T23:21:56.865962Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:04.000000Z", "last_refresh": "2022-03-08T23:21:56.865962Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:24.000000Z", "last_refresh": "2022-03-08T23:21:56.865962Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:07:27.000000Z", "last_refresh": "2022-03-08T23:21:56.865962Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:21:56.865962Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:21:56.865962Z", "running": 0, "size": 1}}] 2022-03-08T23:22:07.046 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph orch ls -f json 2022-03-08T23:22:07.802 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-03-08T23:22:07.802 INFO:teuthology.orchestra.run.smithi077.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:16.000000Z", "last_refresh": "2022-03-08T23:22:07.587048Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:04.000000Z", "last_refresh": "2022-03-08T23:22:07.587048Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:24.000000Z", "last_refresh": "2022-03-08T23:22:07.587048Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:07:27.000000Z", "last_refresh": "2022-03-08T23:22:07.587048Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:22:07.587048Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:22:07.587048Z", "running": 0, "size": 1}}] 2022-03-08T23:22:17.831 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph orch ls -f json 2022-03-08T23:22:18.532 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-03-08T23:22:18.532 INFO:teuthology.orchestra.run.smithi077.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:16.000000Z", "last_refresh": "2022-03-08T23:22:18.354206Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:04.000000Z", "last_refresh": "2022-03-08T23:22:18.354206Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:24.000000Z", "last_refresh": "2022-03-08T23:22:18.354206Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:07:27.000000Z", "last_refresh": "2022-03-08T23:22:18.354206Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:22:18.354206Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:22:18.354206Z", "running": 0, "size": 1}}] 2022-03-08T23:22:28.557 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph orch ls -f json 2022-03-08T23:22:29.346 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-03-08T23:22:29.347 INFO:teuthology.orchestra.run.smithi077.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:16.000000Z", "last_refresh": "2022-03-08T23:22:29.189486Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:04.000000Z", "last_refresh": "2022-03-08T23:22:29.189486Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:24.000000Z", "last_refresh": "2022-03-08T23:22:29.189486Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:07:27.000000Z", "last_refresh": "2022-03-08T23:22:29.189486Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:22:29.189486Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:22:29.189486Z", "running": 0, "size": 1}}] 2022-03-08T23:22:39.371 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph orch ls -f json 2022-03-08T23:22:40.087 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-03-08T23:22:40.087 INFO:teuthology.orchestra.run.smithi077.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:16.000000Z", "last_refresh": "2022-03-08T23:22:39.912377Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:04.000000Z", "last_refresh": "2022-03-08T23:22:39.912377Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:24.000000Z", "last_refresh": "2022-03-08T23:22:39.912377Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:07:27.000000Z", "last_refresh": "2022-03-08T23:22:39.912377Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:22:39.912377Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:22:39.912377Z", "running": 0, "size": 1}}] 2022-03-08T23:22:50.143 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph orch ls -f json 2022-03-08T23:22:50.936 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-03-08T23:22:50.936 INFO:teuthology.orchestra.run.smithi077.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:16.000000Z", "last_refresh": "2022-03-08T23:22:50.687795Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:04.000000Z", "last_refresh": "2022-03-08T23:22:50.687795Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:24.000000Z", "last_refresh": "2022-03-08T23:22:50.687795Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:07:27.000000Z", "last_refresh": "2022-03-08T23:22:50.687795Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:22:50.687795Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:22:50.687795Z", "running": 0, "size": 1}}] 2022-03-08T23:23:00.957 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph orch ls -f json 2022-03-08T23:23:01.661 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-03-08T23:23:01.661 INFO:teuthology.orchestra.run.smithi077.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:16.000000Z", "last_refresh": "2022-03-08T23:23:01.495480Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:04.000000Z", "last_refresh": "2022-03-08T23:23:01.495480Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:24.000000Z", "last_refresh": "2022-03-08T23:23:01.495480Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:07:27.000000Z", "last_refresh": "2022-03-08T23:23:01.495480Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:23:01.495480Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:23:01.495480Z", "running": 0, "size": 1}}] 2022-03-08T23:23:10.958 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:23:10.958835 I | op-mon: parsing mon endpoints: a=10.110.141.123:6789,b=10.98.41.230:6789,c=10.109.47.142:6789 2022-03-08T23:23:11.618 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:23:11.618313 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T23:23:11.608+0000 7f2d37f43380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T23:23:11.618 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08T23:23:11.608+0000 7f2d37f43380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T23:23:11.618 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08T23:23:11.608+0000 7f2d37f43380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T23:23:11.618 INFO:tasks.rook.operator.smithi077.stdout:couldn't init storage provider. : exit status 5 2022-03-08T23:23:11.684 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph orch ls -f json 2022-03-08T23:23:12.350 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-03-08T23:23:12.351 INFO:teuthology.orchestra.run.smithi077.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:16.000000Z", "last_refresh": "2022-03-08T23:23:12.185107Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:04.000000Z", "last_refresh": "2022-03-08T23:23:12.185107Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:24.000000Z", "last_refresh": "2022-03-08T23:23:12.185107Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:07:27.000000Z", "last_refresh": "2022-03-08T23:23:12.185107Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:23:12.185107Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:23:12.185107Z", "running": 0, "size": 1}}] 2022-03-08T23:23:22.374 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph orch ls -f json 2022-03-08T23:23:23.074 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-03-08T23:23:23.075 INFO:teuthology.orchestra.run.smithi077.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:16.000000Z", "last_refresh": "2022-03-08T23:23:22.919602Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:04.000000Z", "last_refresh": "2022-03-08T23:23:22.919602Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:24.000000Z", "last_refresh": "2022-03-08T23:23:22.919602Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:07:27.000000Z", "last_refresh": "2022-03-08T23:23:22.919602Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:23:22.919602Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:23:22.919602Z", "running": 0, "size": 1}}] 2022-03-08T23:23:33.097 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph orch ls -f json 2022-03-08T23:23:33.886 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-03-08T23:23:33.887 INFO:teuthology.orchestra.run.smithi077.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:16.000000Z", "last_refresh": "2022-03-08T23:23:33.658324Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:04.000000Z", "last_refresh": "2022-03-08T23:23:33.658324Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:24.000000Z", "last_refresh": "2022-03-08T23:23:33.658324Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:07:27.000000Z", "last_refresh": "2022-03-08T23:23:33.658324Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:23:33.658324Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:23:33.658324Z", "running": 0, "size": 1}}] 2022-03-08T23:23:43.908 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph orch ls -f json 2022-03-08T23:23:44.701 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-03-08T23:23:44.702 INFO:teuthology.orchestra.run.smithi077.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:16.000000Z", "last_refresh": "2022-03-08T23:23:44.455815Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:04.000000Z", "last_refresh": "2022-03-08T23:23:44.455815Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:24.000000Z", "last_refresh": "2022-03-08T23:23:44.455815Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:07:27.000000Z", "last_refresh": "2022-03-08T23:23:44.455815Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:23:44.455815Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:23:44.455815Z", "running": 0, "size": 1}}] 2022-03-08T23:23:54.722 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph orch ls -f json 2022-03-08T23:23:55.409 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-03-08T23:23:55.410 INFO:teuthology.orchestra.run.smithi077.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:16.000000Z", "last_refresh": "2022-03-08T23:23:55.242736Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:04.000000Z", "last_refresh": "2022-03-08T23:23:55.242736Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:24.000000Z", "last_refresh": "2022-03-08T23:23:55.242736Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:07:27.000000Z", "last_refresh": "2022-03-08T23:23:55.242736Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:23:55.242736Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:23:55.242736Z", "running": 0, "size": 1}}] 2022-03-08T23:24:05.444 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph orch ls -f json 2022-03-08T23:24:06.145 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-03-08T23:24:06.146 INFO:teuthology.orchestra.run.smithi077.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:16.000000Z", "last_refresh": "2022-03-08T23:24:05.980656Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:04.000000Z", "last_refresh": "2022-03-08T23:24:05.980656Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:24.000000Z", "last_refresh": "2022-03-08T23:24:05.980656Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:07:27.000000Z", "last_refresh": "2022-03-08T23:24:05.980656Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:24:05.980656Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:24:05.980656Z", "running": 0, "size": 1}}] 2022-03-08T23:24:16.167 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph orch ls -f json 2022-03-08T23:24:16.889 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-03-08T23:24:16.889 INFO:teuthology.orchestra.run.smithi077.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:16.000000Z", "last_refresh": "2022-03-08T23:24:16.723001Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:04.000000Z", "last_refresh": "2022-03-08T23:24:16.723001Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:24.000000Z", "last_refresh": "2022-03-08T23:24:16.723001Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:07:27.000000Z", "last_refresh": "2022-03-08T23:24:16.723001Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:24:16.723001Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:24:16.723001Z", "running": 0, "size": 1}}] 2022-03-08T23:24:26.909 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph orch ls -f json 2022-03-08T23:24:27.621 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-03-08T23:24:27.621 INFO:teuthology.orchestra.run.smithi077.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:16.000000Z", "last_refresh": "2022-03-08T23:24:27.441917Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:04.000000Z", "last_refresh": "2022-03-08T23:24:27.441917Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:24.000000Z", "last_refresh": "2022-03-08T23:24:27.441917Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:07:27.000000Z", "last_refresh": "2022-03-08T23:24:27.441917Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:24:27.441917Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:24:27.441917Z", "running": 0, "size": 1}}] 2022-03-08T23:24:37.643 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph orch ls -f json 2022-03-08T23:24:38.355 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-03-08T23:24:38.355 INFO:teuthology.orchestra.run.smithi077.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:16.000000Z", "last_refresh": "2022-03-08T23:24:38.204976Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:04.000000Z", "last_refresh": "2022-03-08T23:24:38.204976Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:24.000000Z", "last_refresh": "2022-03-08T23:24:38.204976Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:07:27.000000Z", "last_refresh": "2022-03-08T23:24:38.204976Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:24:38.204976Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:24:38.204976Z", "running": 0, "size": 1}}] 2022-03-08T23:24:48.377 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph orch ls -f json 2022-03-08T23:24:49.046 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-03-08T23:24:49.046 INFO:teuthology.orchestra.run.smithi077.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:16.000000Z", "last_refresh": "2022-03-08T23:24:48.890594Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:04.000000Z", "last_refresh": "2022-03-08T23:24:48.890594Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:24.000000Z", "last_refresh": "2022-03-08T23:24:48.890594Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:07:27.000000Z", "last_refresh": "2022-03-08T23:24:48.890594Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:24:48.890594Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:24:48.890594Z", "running": 0, "size": 1}}] 2022-03-08T23:24:59.073 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph orch ls -f json 2022-03-08T23:24:59.807 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-03-08T23:24:59.808 INFO:teuthology.orchestra.run.smithi077.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:16.000000Z", "last_refresh": "2022-03-08T23:24:59.606135Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:04.000000Z", "last_refresh": "2022-03-08T23:24:59.606135Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:24.000000Z", "last_refresh": "2022-03-08T23:24:59.606135Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:07:27.000000Z", "last_refresh": "2022-03-08T23:24:59.606135Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:24:59.606135Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:24:59.606135Z", "running": 0, "size": 1}}] 2022-03-08T23:25:09.831 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph orch ls -f json 2022-03-08T23:25:10.584 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-03-08T23:25:10.584 INFO:teuthology.orchestra.run.smithi077.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:16.000000Z", "last_refresh": "2022-03-08T23:25:10.431415Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:04.000000Z", "last_refresh": "2022-03-08T23:25:10.431415Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:24.000000Z", "last_refresh": "2022-03-08T23:25:10.431415Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:07:27.000000Z", "last_refresh": "2022-03-08T23:25:10.431415Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:25:10.431415Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:25:10.431415Z", "running": 0, "size": 1}}] 2022-03-08T23:25:20.602 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph orch ls -f json 2022-03-08T23:25:21.759 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-03-08T23:25:21.760 INFO:teuthology.orchestra.run.smithi077.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:16.000000Z", "last_refresh": "2022-03-08T23:25:21.260699Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:04.000000Z", "last_refresh": "2022-03-08T23:25:21.260699Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:24.000000Z", "last_refresh": "2022-03-08T23:25:21.260699Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:07:27.000000Z", "last_refresh": "2022-03-08T23:25:21.260699Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:25:21.260699Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:25:21.260699Z", "running": 0, "size": 1}}] 2022-03-08T23:25:31.783 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph orch ls -f json 2022-03-08T23:25:32.906 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-03-08T23:25:32.906 INFO:teuthology.orchestra.run.smithi077.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:16.000000Z", "last_refresh": "2022-03-08T23:25:32.749247Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:04.000000Z", "last_refresh": "2022-03-08T23:25:32.749247Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:24.000000Z", "last_refresh": "2022-03-08T23:25:32.749247Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:07:27.000000Z", "last_refresh": "2022-03-08T23:25:32.749247Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:25:32.749247Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:25:32.749247Z", "running": 0, "size": 1}}] 2022-03-08T23:25:42.929 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph orch ls -f json 2022-03-08T23:25:43.598 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-03-08T23:25:43.598 INFO:teuthology.orchestra.run.smithi077.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:16.000000Z", "last_refresh": "2022-03-08T23:25:43.443006Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:04.000000Z", "last_refresh": "2022-03-08T23:25:43.443006Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:24.000000Z", "last_refresh": "2022-03-08T23:25:43.443006Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:07:27.000000Z", "last_refresh": "2022-03-08T23:25:43.443006Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:25:43.443006Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:25:43.443006Z", "running": 0, "size": 1}}] 2022-03-08T23:25:53.622 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph orch ls -f json 2022-03-08T23:25:54.377 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-03-08T23:25:54.377 INFO:teuthology.orchestra.run.smithi077.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:16.000000Z", "last_refresh": "2022-03-08T23:25:54.159013Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:04.000000Z", "last_refresh": "2022-03-08T23:25:54.159013Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:24.000000Z", "last_refresh": "2022-03-08T23:25:54.159013Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:07:27.000000Z", "last_refresh": "2022-03-08T23:25:54.159013Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:25:54.159013Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:25:54.159013Z", "running": 0, "size": 1}}] 2022-03-08T23:26:04.405 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph orch ls -f json 2022-03-08T23:26:05.140 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-03-08T23:26:05.141 INFO:teuthology.orchestra.run.smithi077.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:16.000000Z", "last_refresh": "2022-03-08T23:26:04.976465Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:04.000000Z", "last_refresh": "2022-03-08T23:26:04.976465Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:24.000000Z", "last_refresh": "2022-03-08T23:26:04.976465Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:07:27.000000Z", "last_refresh": "2022-03-08T23:26:04.976465Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:26:04.976465Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:26:04.976465Z", "running": 0, "size": 1}}] 2022-03-08T23:26:15.165 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph orch ls -f json 2022-03-08T23:26:15.895 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-03-08T23:26:15.896 INFO:teuthology.orchestra.run.smithi077.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:16.000000Z", "last_refresh": "2022-03-08T23:26:15.732934Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:04.000000Z", "last_refresh": "2022-03-08T23:26:15.732934Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:24.000000Z", "last_refresh": "2022-03-08T23:26:15.732934Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:07:27.000000Z", "last_refresh": "2022-03-08T23:26:15.732934Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:26:15.732934Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:26:15.732934Z", "running": 0, "size": 1}}] 2022-03-08T23:26:25.918 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph orch ls -f json 2022-03-08T23:26:26.658 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-03-08T23:26:26.658 INFO:teuthology.orchestra.run.smithi077.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:16.000000Z", "last_refresh": "2022-03-08T23:26:26.467065Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:04.000000Z", "last_refresh": "2022-03-08T23:26:26.467065Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:24.000000Z", "last_refresh": "2022-03-08T23:26:26.467065Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:07:27.000000Z", "last_refresh": "2022-03-08T23:26:26.467065Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:26:26.467065Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:26:26.467065Z", "running": 0, "size": 1}}] 2022-03-08T23:26:36.687 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph orch ls -f json 2022-03-08T23:26:37.367 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-03-08T23:26:37.367 INFO:teuthology.orchestra.run.smithi077.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:16.000000Z", "last_refresh": "2022-03-08T23:26:37.213403Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:04.000000Z", "last_refresh": "2022-03-08T23:26:37.213403Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:24.000000Z", "last_refresh": "2022-03-08T23:26:37.213403Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:07:27.000000Z", "last_refresh": "2022-03-08T23:26:37.213403Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:26:37.213403Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:26:37.213403Z", "running": 0, "size": 1}}] 2022-03-08T23:26:47.389 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph orch ls -f json 2022-03-08T23:26:48.061 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-03-08T23:26:48.062 INFO:teuthology.orchestra.run.smithi077.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:16.000000Z", "last_refresh": "2022-03-08T23:26:47.903674Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:04.000000Z", "last_refresh": "2022-03-08T23:26:47.903674Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:24.000000Z", "last_refresh": "2022-03-08T23:26:47.903674Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:07:27.000000Z", "last_refresh": "2022-03-08T23:26:47.903674Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:26:47.903674Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:26:47.903674Z", "running": 0, "size": 1}}] 2022-03-08T23:26:58.083 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph orch ls -f json 2022-03-08T23:26:58.769 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-03-08T23:26:58.770 INFO:teuthology.orchestra.run.smithi077.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:16.000000Z", "last_refresh": "2022-03-08T23:26:58.617776Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:04.000000Z", "last_refresh": "2022-03-08T23:26:58.617776Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:24.000000Z", "last_refresh": "2022-03-08T23:26:58.617776Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:07:27.000000Z", "last_refresh": "2022-03-08T23:26:58.617776Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:26:58.617776Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:26:58.617776Z", "running": 0, "size": 1}}] 2022-03-08T23:27:08.788 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph orch ls -f json 2022-03-08T23:27:09.500 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-03-08T23:27:09.500 INFO:teuthology.orchestra.run.smithi077.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:16.000000Z", "last_refresh": "2022-03-08T23:27:09.303798Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:04.000000Z", "last_refresh": "2022-03-08T23:27:09.303798Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:24.000000Z", "last_refresh": "2022-03-08T23:27:09.303798Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:07:27.000000Z", "last_refresh": "2022-03-08T23:27:09.303798Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:27:09.303798Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:27:09.303798Z", "running": 0, "size": 1}}] 2022-03-08T23:27:19.523 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph orch ls -f json 2022-03-08T23:27:20.313 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-03-08T23:27:20.314 INFO:teuthology.orchestra.run.smithi077.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:16.000000Z", "last_refresh": "2022-03-08T23:27:20.101670Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:04.000000Z", "last_refresh": "2022-03-08T23:27:20.101670Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:24.000000Z", "last_refresh": "2022-03-08T23:27:20.101670Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:07:27.000000Z", "last_refresh": "2022-03-08T23:27:20.101670Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:27:20.101670Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:27:20.101670Z", "running": 0, "size": 1}}] 2022-03-08T23:27:30.337 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph orch ls -f json 2022-03-08T23:27:31.088 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-03-08T23:27:31.089 INFO:teuthology.orchestra.run.smithi077.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:16.000000Z", "last_refresh": "2022-03-08T23:27:30.879018Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:04.000000Z", "last_refresh": "2022-03-08T23:27:30.879018Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:24.000000Z", "last_refresh": "2022-03-08T23:27:30.879018Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:07:27.000000Z", "last_refresh": "2022-03-08T23:27:30.879018Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:27:30.879018Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:27:30.879018Z", "running": 0, "size": 1}}] 2022-03-08T23:27:41.110 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph orch ls -f json 2022-03-08T23:27:41.862 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-03-08T23:27:41.862 INFO:teuthology.orchestra.run.smithi077.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:16.000000Z", "last_refresh": "2022-03-08T23:27:41.701345Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:04.000000Z", "last_refresh": "2022-03-08T23:27:41.701345Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:24.000000Z", "last_refresh": "2022-03-08T23:27:41.701345Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:07:27.000000Z", "last_refresh": "2022-03-08T23:27:41.701345Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:27:41.701345Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:27:41.701345Z", "running": 0, "size": 1}}] 2022-03-08T23:27:51.881 DEBUG:teuthology.orchestra.run.smithi077:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-vmw72 -- ceph orch ls -f json 2022-03-08T23:27:52.585 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-03-08T23:27:52.585 INFO:teuthology.orchestra.run.smithi077.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:16.000000Z", "last_refresh": "2022-03-08T23:27:52.420404Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:04.000000Z", "last_refresh": "2022-03-08T23:27:52.420404Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:03:24.000000Z", "last_refresh": "2022-03-08T23:27:52.420404Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:07:27.000000Z", "last_refresh": "2022-03-08T23:27:52.420404Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:27:52.420404Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:27:52.420404Z", "running": 0, "size": 1}}] 2022-03-08T23:27:52.609 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/contextutil.py", line 33, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_7a44e9ebca556862e9704be295a07719d7db47b3/qa/tasks/rook.py", line 669, in task while proceed(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds 2022-03-08T23:27:52.610 ERROR:tasks.rook:'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_7a44e9ebca556862e9704be295a07719d7db47b3/qa/tasks/rook.py", line 530, in ceph_config_keyring yield File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/contextutil.py", line 33, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_7a44e9ebca556862e9704be295a07719d7db47b3/qa/tasks/rook.py", line 669, in task while proceed(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds 2022-03-08T23:27:52.610 INFO:tasks.rook:Cleaning up config and client.admin keyring 2022-03-08T23:27:52.611 DEBUG:teuthology.orchestra.run.smithi077:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2022-03-08T23:27:52.625 DEBUG:teuthology.orchestra.run.smithi106:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2022-03-08T23:27:52.640 DEBUG:teuthology.orchestra.run.smithi135:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2022-03-08T23:27:52.655 ERROR:tasks.rook:'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_7a44e9ebca556862e9704be295a07719d7db47b3/qa/tasks/rook.py", line 478, in rook_post_config yield File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/contextutil.py", line 33, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_7a44e9ebca556862e9704be295a07719d7db47b3/qa/tasks/rook.py", line 669, in task while proceed(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds 2022-03-08T23:27:52.656 ERROR:tasks.rook:'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_7a44e9ebca556862e9704be295a07719d7db47b3/qa/tasks/rook.py", line 442, in rook_toolbox yield File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/contextutil.py", line 33, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_7a44e9ebca556862e9704be295a07719d7db47b3/qa/tasks/rook.py", line 669, in task while proceed(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds 2022-03-08T23:27:52.670 DEBUG:teuthology.orchestra.remote:smithi077:rook/deploy/examples/operator.yaml is 20KB 2022-03-08T23:27:52.685 DEBUG:teuthology.orchestra.run.smithi077:> kubectl delete -f rook/deploy/examples/toolbox.yaml 2022-03-08T23:27:52.754 INFO:teuthology.orchestra.run.smithi077.stdout:deployment.apps "rook-ceph-tools" deleted 2022-03-08T23:27:52.778 ERROR:tasks.rook:'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_7a44e9ebca556862e9704be295a07719d7db47b3/qa/tasks/rook.py", line 379, in rook_cluster yield File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/contextutil.py", line 33, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_7a44e9ebca556862e9704be295a07719d7db47b3/qa/tasks/rook.py", line 669, in task while proceed(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds 2022-03-08T23:27:52.778 DEBUG:teuthology.orchestra.run.smithi077:> kubectl delete -f cluster.yaml 2022-03-08T23:27:52.882 INFO:teuthology.orchestra.run.smithi077.stdout:cephcluster.ceph.rook.io "rook-ceph" deleted 2022-03-08T23:27:52.884 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:27:52.885059 I | ceph-cluster-controller: CR "rook-ceph" is going be deleted, cancelling any ongoing orchestration 2022-03-08T23:27:52.885 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:27:52.885148 I | operator: reloading operator's CRDs manager, cancelling all orchestrations! 2022-03-08T23:27:52.885 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:27:52.885424 I | operator: successfully started the controller-runtime manager 2022-03-08T23:27:52.885 INFO:tasks.rook.operator.smithi077.stdout:I0308 23:27:52.885533 1 manager.go:148] objectbucket.io/provisioner-manager "msg"="stopping provisioner" "name"="rook-ceph.ceph.rook.io/bucket" "reason"={} 2022-03-08T23:27:52.905 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:27:52.905536 I | op-k8sutil: ROOK_CURRENT_NAMESPACE_ONLY="false" (env var) 2022-03-08T23:27:52.905 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:27:52.905564 I | operator: watching all namespaces for Ceph CRs 2022-03-08T23:27:52.906 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:27:52.905609 I | operator: setting up schemes 2022-03-08T23:27:52.912 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:27:52.909176 I | operator: setting up the controller-runtime manager 2022-03-08T23:27:53.113 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:27:53.113488 I | operator: looking for admission webhook secret "rook-ceph-admission-controller" 2022-03-08T23:27:53.119 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:27:53.116325 I | operator: admission webhook secret "rook-ceph-admission-controller" not found. proceeding without the admission controller 2022-03-08T23:27:53.119 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:27:53.116368 I | ceph-cluster-controller: successfully started 2022-03-08T23:27:53.120 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:27:53.116472 I | ceph-cluster-controller: enabling hotplug orchestration 2022-03-08T23:27:53.120 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:27:53.116504 I | ceph-crashcollector-controller: successfully started 2022-03-08T23:27:53.120 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:27:53.116532 I | ceph-block-pool-controller: successfully started 2022-03-08T23:27:53.120 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:27:53.116556 I | ceph-object-store-user-controller: successfully started 2022-03-08T23:27:53.120 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:27:53.116586 I | ceph-object-realm-controller: successfully started 2022-03-08T23:27:53.121 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:27:53.116603 I | ceph-object-zonegroup-controller: successfully started 2022-03-08T23:27:53.121 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:27:53.116623 I | ceph-object-zone-controller: successfully started 2022-03-08T23:27:53.121 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:27:53.116791 I | ceph-object-controller: successfully started 2022-03-08T23:27:53.121 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:27:53.116832 I | ceph-file-controller: successfully started 2022-03-08T23:27:53.122 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:27:53.116898 I | ceph-nfs-controller: successfully started 2022-03-08T23:27:53.122 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:27:53.116952 I | ceph-rbd-mirror-controller: successfully started 2022-03-08T23:27:53.122 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:27:53.117010 I | ceph-client-controller: successfully started 2022-03-08T23:27:53.122 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:27:53.117050 I | ceph-filesystem-mirror-controller: successfully started 2022-03-08T23:27:53.123 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:27:53.117104 I | operator: rook-ceph-operator-config-controller successfully started 2022-03-08T23:27:53.123 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:27:53.117151 I | ceph-csi: rook-ceph-operator-csi-controller successfully started 2022-03-08T23:27:53.123 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:27:53.117188 I | op-bucket-prov: rook-ceph-operator-bucket-controller successfully started 2022-03-08T23:27:53.123 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:27:53.117218 I | ceph-bucket-topic: successfully started 2022-03-08T23:27:53.123 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:27:53.117247 I | ceph-bucket-notification: successfully started 2022-03-08T23:27:53.124 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:27:53.117274 I | ceph-bucket-notification: successfully started 2022-03-08T23:27:53.124 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:27:53.117297 I | ceph-fs-subvolumegroup-controller: successfully started 2022-03-08T23:27:53.124 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:27:53.118661 I | operator: starting the controller-runtime manager 2022-03-08T23:27:53.224 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:27:53.224133 I | op-k8sutil: ROOK_CEPH_COMMANDS_TIMEOUT_SECONDS="15" (configmap) 2022-03-08T23:27:53.224 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:27:53.224191 I | op-k8sutil: ROOK_LOG_LEVEL="INFO" (configmap) 2022-03-08T23:27:53.225 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:27:53.224204 I | op-k8sutil: ROOK_ENABLE_DISCOVERY_DAEMON="false" (configmap) 2022-03-08T23:27:53.228 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:27:53.228096 I | operator: rook-ceph-operator-config-controller done reconciling 2022-03-08T23:27:53.228 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:27:53.228203 I | op-mon: parsing mon endpoints: a=10.110.141.123:6789,b=10.98.41.230:6789,c=10.109.47.142:6789 2022-03-08T23:27:53.296 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:27:53.296931 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:27:53.355 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:27:53.355830 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:27:53.884 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:27:53.884527 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T23:27:53.873+0000 7fb2f99cf380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T23:27:53.885 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08T23:27:53.873+0000 7fb2f99cf380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T23:27:53.885 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08T23:27:53.873+0000 7fb2f99cf380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T23:27:53.885 INFO:tasks.rook.operator.smithi077.stdout:couldn't init storage provider. : exit status 5 2022-03-08T23:27:53.904 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:27:53.904632 I | op-mon: parsing mon endpoints: a=10.110.141.123:6789,b=10.98.41.230:6789,c=10.109.47.142:6789 2022-03-08T23:27:54.622 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:27:54.622747 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T23:27:54.613+0000 7fe9d4a79380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T23:27:54.623 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08T23:27:54.613+0000 7fe9d4a79380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T23:27:54.623 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08T23:27:54.613+0000 7fe9d4a79380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T23:27:54.623 INFO:tasks.rook.operator.smithi077.stdout:couldn't init storage provider. : exit status 5 2022-03-08T23:27:54.637 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:27:54.638413 I | op-mon: parsing mon endpoints: a=10.110.141.123:6789,b=10.98.41.230:6789,c=10.109.47.142:6789 2022-03-08T23:27:55.274 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:27:55.273951 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T23:27:55.264+0000 7f8129dd9380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T23:27:55.274 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08T23:27:55.264+0000 7f8129dd9380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T23:27:55.274 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08T23:27:55.264+0000 7f8129dd9380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T23:27:55.275 INFO:tasks.rook.operator.smithi077.stdout:couldn't init storage provider. : exit status 5 2022-03-08T23:27:55.300 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:27:55.300927 I | op-mon: parsing mon endpoints: a=10.110.141.123:6789,b=10.98.41.230:6789,c=10.109.47.142:6789 2022-03-08T23:27:55.979 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:27:55.979475 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T23:27:55.968+0000 7f1fdc755380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T23:27:55.979 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08T23:27:55.968+0000 7f1fdc755380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T23:27:55.980 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08T23:27:55.968+0000 7f1fdc755380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T23:27:55.980 INFO:tasks.rook.operator.smithi077.stdout:couldn't init storage provider. : exit status 5 2022-03-08T23:27:56.027 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:27:56.027516 I | op-mon: parsing mon endpoints: a=10.110.141.123:6789,b=10.98.41.230:6789,c=10.109.47.142:6789 2022-03-08T23:27:56.676 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:27:56.676879 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T23:27:56.664+0000 7f47af3cd380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T23:27:56.677 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08T23:27:56.664+0000 7f47af3cd380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T23:27:56.677 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08T23:27:56.664+0000 7f47af3cd380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T23:27:56.677 INFO:tasks.rook.operator.smithi077.stdout:couldn't init storage provider. : exit status 5 2022-03-08T23:27:56.778 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:27:56.778583 I | op-mon: parsing mon endpoints: a=10.110.141.123:6789,b=10.98.41.230:6789,c=10.109.47.142:6789 2022-03-08T23:27:57.621 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:27:57.621286 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T23:27:57.608+0000 7f88ad9e9380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T23:27:57.621 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08T23:27:57.608+0000 7f88ad9e9380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T23:27:57.622 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08T23:27:57.608+0000 7f88ad9e9380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T23:27:57.622 INFO:tasks.rook.operator.smithi077.stdout:couldn't init storage provider. : exit status 5 2022-03-08T23:27:57.789 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:27:57.789904 I | op-mon: parsing mon endpoints: a=10.110.141.123:6789,b=10.98.41.230:6789,c=10.109.47.142:6789 2022-03-08T23:27:58.425 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:27:58.425422 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T23:27:58.416+0000 7f4516511380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T23:27:58.425 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08T23:27:58.416+0000 7f4516511380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T23:27:58.426 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08T23:27:58.416+0000 7f4516511380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T23:27:58.426 INFO:tasks.rook.operator.smithi077.stdout:couldn't init storage provider. : exit status 5 2022-03-08T23:27:58.752 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:27:58.752586 I | op-mon: parsing mon endpoints: a=10.110.141.123:6789,b=10.98.41.230:6789,c=10.109.47.142:6789 2022-03-08T23:27:59.412 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:27:59.412364 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T23:27:59.400+0000 7f0ed7dd0380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T23:27:59.413 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08T23:27:59.400+0000 7f0ed7dd0380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T23:27:59.413 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08T23:27:59.400+0000 7f0ed7dd0380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T23:27:59.414 INFO:tasks.rook.operator.smithi077.stdout:couldn't init storage provider. : exit status 5 2022-03-08T23:28:00.069 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:28:00.069975 I | op-mon: parsing mon endpoints: a=10.110.141.123:6789,b=10.98.41.230:6789,c=10.109.47.142:6789 2022-03-08T23:28:00.736 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:28:00.736501 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T23:28:00.724+0000 7ff4a643a380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T23:28:00.736 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08T23:28:00.724+0000 7ff4a643a380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T23:28:00.736 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08T23:28:00.724+0000 7ff4a643a380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T23:28:00.737 INFO:tasks.rook.operator.smithi077.stdout:couldn't init storage provider. : exit status 5 2022-03-08T23:28:02.024 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:28:02.024421 I | op-mon: parsing mon endpoints: a=10.110.141.123:6789,b=10.98.41.230:6789,c=10.109.47.142:6789 2022-03-08T23:28:02.670 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:28:02.670234 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T23:28:02.660+0000 7f42e575c380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T23:28:02.670 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08T23:28:02.660+0000 7f42e575c380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T23:28:02.670 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08T23:28:02.660+0000 7f42e575c380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T23:28:02.671 INFO:tasks.rook.operator.smithi077.stdout:couldn't init storage provider. : exit status 5 2022-03-08T23:28:03.460 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:28:03.460580 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:28:03.472 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:28:03.472864 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:28:05.251 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:28:05.252230 I | op-mon: parsing mon endpoints: a=10.110.141.123:6789,b=10.98.41.230:6789,c=10.109.47.142:6789 2022-03-08T23:28:05.884 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:28:05.884294 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T23:28:05.872+0000 7f06151c0380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T23:28:05.884 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08T23:28:05.872+0000 7f06151c0380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T23:28:05.885 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08T23:28:05.872+0000 7f06151c0380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T23:28:05.885 INFO:tasks.rook.operator.smithi077.stdout:couldn't init storage provider. : exit status 5 2022-03-08T23:28:11.010 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:28:11.011403 I | op-mon: parsing mon endpoints: a=10.110.141.123:6789,b=10.98.41.230:6789,c=10.109.47.142:6789 2022-03-08T23:28:11.656 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:28:11.656661 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T23:28:11.644+0000 7fd1bb946380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T23:28:11.657 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08T23:28:11.644+0000 7fd1bb946380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T23:28:11.657 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08T23:28:11.644+0000 7fd1bb946380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T23:28:11.657 INFO:tasks.rook.operator.smithi077.stdout:couldn't init storage provider. : exit status 5 2022-03-08T23:28:13.584 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:28:13.584869 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:28:13.594 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:28:13.595473 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:28:21.910 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:28:21.910783 I | op-mon: parsing mon endpoints: a=10.110.141.123:6789,b=10.98.41.230:6789,c=10.109.47.142:6789 2022-03-08T23:28:22.575 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:28:22.575134 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T23:28:22.564+0000 7f578fd8e380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T23:28:22.575 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08T23:28:22.564+0000 7f578fd8e380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T23:28:22.575 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08T23:28:22.564+0000 7f578fd8e380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T23:28:22.576 INFO:tasks.rook.operator.smithi077.stdout:couldn't init storage provider. : exit status 5 2022-03-08T23:28:23.677 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:28:23.677808 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:28:23.690 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:28:23.690837 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:28:33.796 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:28:33.796394 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:28:33.807 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:28:33.808546 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:28:43.062 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:28:43.062455 I | op-mon: parsing mon endpoints: a=10.110.141.123:6789,b=10.98.41.230:6789,c=10.109.47.142:6789 2022-03-08T23:28:43.714 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:28:43.714118 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T23:28:43.703+0000 7f8cb1741380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T23:28:43.714 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08T23:28:43.703+0000 7f8cb1741380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T23:28:43.714 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08T23:28:43.703+0000 7f8cb1741380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T23:28:43.715 INFO:tasks.rook.operator.smithi077.stdout:couldn't init storage provider. : exit status 5 2022-03-08T23:28:43.915 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:28:43.915927 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:28:43.927 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:28:43.927843 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:28:54.039 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:28:54.039390 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:28:54.051 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:28:54.052285 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:29:04.169 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:29:04.170053 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:29:04.184 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:29:04.184910 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:29:14.303 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:29:14.303398 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:29:14.316 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:29:14.316733 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:29:24.388 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:29:24.388656 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:29:24.403 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:29:24.403752 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:29:24.680 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:29:24.680707 I | op-mon: parsing mon endpoints: a=10.110.141.123:6789,b=10.98.41.230:6789,c=10.109.47.142:6789 2022-03-08T23:29:25.339 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:29:25.339572 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T23:29:25.330+0000 7fc41817b380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T23:29:25.340 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08T23:29:25.330+0000 7fc41817b380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T23:29:25.340 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08T23:29:25.330+0000 7fc41817b380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T23:29:25.340 INFO:tasks.rook.operator.smithi077.stdout:couldn't init storage provider. : exit status 5 2022-03-08T23:29:34.493 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:29:34.493051 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:29:34.551 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:29:34.551299 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:29:44.620 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:29:44.619595 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:29:44.633 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:29:44.633996 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:29:54.749 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:29:54.748983 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:29:54.771 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:29:54.771703 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:30:04.970 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:30:04.970055 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:30:04.982 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:30:04.982774 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:30:15.070 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:30:15.070149 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:30:15.120 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:30:15.120695 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:30:25.189 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:30:25.189901 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:30:25.203 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:30:25.203161 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:30:35.319 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:30:35.319451 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:30:35.330 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:30:35.330712 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:30:45.391 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:30:45.391524 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:30:45.403 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:30:45.403405 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:30:47.281 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:30:47.281353 I | op-mon: parsing mon endpoints: a=10.110.141.123:6789,b=10.98.41.230:6789,c=10.109.47.142:6789 2022-03-08T23:30:47.935 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:30:47.934745 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T23:30:47.924+0000 7fd8189e5380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T23:30:47.935 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08T23:30:47.924+0000 7fd8189e5380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T23:30:47.935 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08T23:30:47.924+0000 7fd8189e5380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T23:30:47.935 INFO:tasks.rook.operator.smithi077.stdout:couldn't init storage provider. : exit status 5 2022-03-08T23:30:55.481 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:30:55.481510 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:30:55.495 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:30:55.495488 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:31:05.554 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:31:05.554255 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:31:05.565 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:31:05.565693 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:31:15.664 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:31:15.664474 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:31:15.683 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:31:15.683882 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:31:25.825 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:31:25.825390 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:31:25.837 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:31:25.837214 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:31:35.931 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:31:35.931801 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:31:35.942 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:31:35.943031 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:31:46.095 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:31:46.095845 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:31:46.108 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:31:46.108943 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:31:56.221 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:31:56.220974 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:31:56.238 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:31:56.238882 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:32:06.302 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:32:06.302650 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:32:06.365 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:32:06.365510 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:32:16.475 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:32:16.475181 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:32:16.484 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:32:16.484535 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:32:26.545 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:32:26.545831 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:32:26.598 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:32:26.598142 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:32:36.699 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:32:36.699934 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:32:36.710 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:32:36.710764 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:32:46.802 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:32:46.802059 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:32:46.814 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:32:46.814596 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:32:56.893 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:32:56.893585 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:32:56.904 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:32:56.904525 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:33:07.003 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:33:07.003137 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:33:07.013 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:33:07.013982 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:33:17.113 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:33:17.113228 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:33:17.136 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:33:17.136954 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:33:27.255 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:33:27.255688 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:33:27.269 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:33:27.269547 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:33:31.807 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:33:31.807778 I | op-mon: parsing mon endpoints: a=10.110.141.123:6789,b=10.98.41.230:6789,c=10.109.47.142:6789 2022-03-08T23:33:32.438 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:33:32.438256 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T23:33:32.428+0000 7fad0ce0f380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T23:33:32.439 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08T23:33:32.428+0000 7fad0ce0f380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T23:33:32.439 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08T23:33:32.428+0000 7fad0ce0f380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T23:33:32.439 INFO:tasks.rook.operator.smithi077.stdout:couldn't init storage provider. : exit status 5 2022-03-08T23:33:37.333 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:33:37.333599 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:33:37.345 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:33:37.346131 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:33:47.478 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:33:47.478255 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:33:47.492 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:33:47.492657 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:33:57.565 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:33:57.565816 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:33:57.578 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:33:57.578831 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:34:07.647 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:34:07.647718 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:34:07.661 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:34:07.661615 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:34:17.743 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:34:17.742867 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:34:17.756 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:34:17.756739 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:34:27.863 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:34:27.863418 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:34:27.872 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:34:27.872495 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:34:37.989 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:34:37.988832 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:34:38.001 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:34:38.002333 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:34:48.094 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:34:48.094492 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:34:48.106 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:34:48.106809 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:34:58.227 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:34:58.227105 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:34:58.243 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:34:58.243279 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:35:08.355 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:35:08.355281 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:35:08.366 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:35:08.366994 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:35:18.464 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:35:18.463879 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:35:18.475 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:35:18.475444 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:35:28.570 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:35:28.570536 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:35:28.582 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:35:28.582943 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:35:38.650 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:35:38.650235 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:35:38.662 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:35:38.663154 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:35:48.935 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:35:48.935822 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:35:48.946 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:35:48.947144 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:35:59.084 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:35:59.084344 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:35:59.098 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:35:59.099073 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:36:09.205 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:36:09.205788 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:36:09.222 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:36:09.221706 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:36:19.284 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:36:19.282839 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:36:19.297 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:36:19.297502 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:36:29.420 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:36:29.420463 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:36:29.431 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:36:29.432027 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:36:39.523 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:36:39.523409 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:36:39.537 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:36:39.537545 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:36:49.667 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:36:49.667422 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:36:49.681 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:36:49.681525 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:36:59.814 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:36:59.814548 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:36:59.826 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:36:59.827524 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:37:09.897 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:37:09.897349 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:37:09.954 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:37:09.954764 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:37:20.027 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:37:20.027085 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:37:20.036 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:37:20.037097 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:37:30.145 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:37:30.145738 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:37:30.157 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:37:30.157847 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:37:40.262 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:37:40.262638 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:37:40.275 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:37:40.275456 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:37:50.353 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:37:50.353186 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:37:50.427 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:37:50.427358 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:38:00.548 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:38:00.548816 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:38:00.562 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:38:00.562923 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:38:10.646 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:38:10.647029 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:38:10.657 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:38:10.657679 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:38:20.719 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:38:20.719556 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:38:20.732 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:38:20.732196 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:38:30.845 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:38:30.844915 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:38:30.858 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:38:30.859046 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:38:40.935 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:38:40.935513 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:38:41.394 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:38:40.947693 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:38:51.023 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:38:51.023614 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:38:51.036 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:38:51.036426 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:39:00.129 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:39:00.129261 I | op-mon: parsing mon endpoints: a=10.110.141.123:6789,b=10.98.41.230:6789,c=10.109.47.142:6789 2022-03-08T23:39:00.812 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:39:00.812329 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T23:39:00.800+0000 7f388c7fe380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T23:39:00.813 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08T23:39:00.800+0000 7f388c7fe380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T23:39:00.813 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08T23:39:00.800+0000 7f388c7fe380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T23:39:00.813 INFO:tasks.rook.operator.smithi077.stdout:couldn't init storage provider. : exit status 5 2022-03-08T23:39:01.097 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:39:01.097240 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:39:01.109 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:39:01.110274 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:39:11.211 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:39:11.211945 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:39:11.222 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:39:11.223176 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:39:21.314 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:39:21.314047 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:39:21.326 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:39:21.326959 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:39:31.401 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:39:31.401217 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:39:31.414 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:39:31.414229 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:39:41.515 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:39:41.515834 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:39:41.527 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:39:41.527870 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:39:51.605 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:39:51.605420 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:39:51.617 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:39:51.618179 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:40:01.677 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:40:01.675398 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:40:01.688 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:40:01.688604 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:40:11.776 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:40:11.776297 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:40:11.825 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:40:11.825701 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:40:21.900 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:40:21.900510 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:40:21.912 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:40:21.913003 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:40:31.994 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:40:31.994457 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:40:32.007 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:40:32.008122 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:40:42.128 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:40:42.127065 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:40:42.139 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:40:42.139947 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:40:52.198 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:40:52.198981 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:40:52.257 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:40:52.257616 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:41:02.382 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:41:02.382428 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:41:02.396 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:41:02.397171 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:41:12.502 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:41:12.502167 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:41:12.535 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:41:12.536328 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:41:22.660 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:41:22.660682 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:41:22.673 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:41:22.673412 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:41:32.810 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:41:32.810554 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:41:32.821 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:41:32.821888 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:41:42.927 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:41:42.927297 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:41:42.942 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:41:42.942505 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:41:53.048 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:41:53.048191 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:41:53.060 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:41:53.060613 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:42:03.195 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:42:03.195594 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:42:03.208 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:42:03.208012 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:42:13.317 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:42:13.317439 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:42:13.334 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:42:13.335382 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:42:23.430 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:42:23.430786 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:42:23.442 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:42:23.443090 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:42:33.531 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:42:33.531533 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:42:33.546 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:42:33.546962 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:42:43.618 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:42:43.618624 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:42:43.630 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:42:43.631158 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:42:53.687 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:42:53.687552 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:42:53.702 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:42:53.702816 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:43:03.783 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:43:03.783452 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:43:03.796 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:43:03.797212 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:43:13.951 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:43:13.950920 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:43:13.964 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:43:13.964787 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:43:24.038 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:43:24.039037 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:43:24.105 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:43:24.105855 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:43:34.266 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:43:34.265903 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:43:34.300 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:43:34.300945 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:43:44.389 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:43:44.390115 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:43:44.401 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:43:44.401988 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:43:54.490 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:43:54.490678 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:43:54.503 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:43:54.503478 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:44:04.630 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:44:04.630559 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:44:04.640 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:44:04.641197 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:44:14.716 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:44:14.716274 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:44:14.731 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:44:14.731228 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:44:24.853 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:44:24.853446 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:44:24.862 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:44:24.863402 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:44:34.937 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:44:34.937469 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:44:34.947 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:44:34.948250 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:44:45.058 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:44:45.059001 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:44:45.071 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:44:45.071645 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:44:55.200 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:44:55.200989 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:44:55.213 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:44:55.213219 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:45:05.345 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:45:05.345716 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:45:05.355 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:45:05.356190 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:45:15.463 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:45:15.463948 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:45:15.474 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:45:15.475244 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:45:25.600 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:45:25.600492 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:45:25.609 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:45:25.610355 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:45:35.702 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:45:35.703121 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:45:35.714 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:45:35.714835 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:45:45.780 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:45:45.780440 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:45:45.790 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:45:45.790952 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:45:55.896 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:45:55.896218 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:45:55.907 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:45:55.907603 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:46:06.023 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:46:06.023448 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:46:06.039 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:46:06.039944 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:46:16.179 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:46:16.179640 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:46:16.192 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:46:16.193160 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:46:26.265 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:46:26.265121 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:46:26.277 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:46:26.278192 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:46:36.341 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:46:36.338191 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:46:36.349 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:46:36.349718 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:46:46.438 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:46:46.438485 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:46:46.450 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:46:46.450930 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:46:56.549 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:46:56.549894 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:46:56.563 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:46:56.563841 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:47:06.631 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:47:06.631705 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:47:06.643 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:47:06.644198 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:47:16.718 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:47:16.717609 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:47:16.732 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:47:16.732754 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:47:26.849 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:47:26.850106 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:47:26.861 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:47:26.861655 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:47:36.971 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:47:36.971545 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:47:36.985 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:47:36.986019 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:47:47.121 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:47:47.122028 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:47:47.135 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:47:47.136224 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:47:57.205 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:47:57.206195 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:47:57.219 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:47:57.220276 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:48:07.357 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:48:07.357814 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:48:07.366 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:48:07.366826 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:48:17.505 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:48:17.505609 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:48:17.516 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:48:17.517144 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:48:27.634 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:48:27.634677 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:48:27.649 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:48:27.650157 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:48:37.925 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:48:37.926124 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:48:37.938 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:48:37.939504 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:48:48.000 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:48:48.001189 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:48:48.010 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:48:48.011090 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:48:58.076 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:48:58.076883 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:48:58.089 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:48:58.090619 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:49:08.220 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:49:08.220771 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:49:08.228 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:49:08.229520 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:49:18.315 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:49:18.315690 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:49:18.326 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:49:18.327161 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:49:28.389 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:49:28.390464 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:49:28.401 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:49:28.402544 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:49:38.496 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:49:38.496791 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:49:38.508 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:49:38.508781 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:49:48.635 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:49:48.636269 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:49:48.651 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:49:48.651629 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:49:56.181 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:49:56.181424 I | op-mon: parsing mon endpoints: a=10.110.141.123:6789,b=10.98.41.230:6789,c=10.109.47.142:6789 2022-03-08T23:49:56.845 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:49:56.844986 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T23:49:56.832+0000 7f26e5062380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T23:49:56.845 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08T23:49:56.832+0000 7f26e5062380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T23:49:56.845 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08T23:49:56.832+0000 7f26e5062380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T23:49:56.845 INFO:tasks.rook.operator.smithi077.stdout:couldn't init storage provider. : exit status 5 2022-03-08T23:49:58.708 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:49:58.709225 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:49:58.722 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:49:58.722925 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:50:08.786 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:50:08.787280 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:50:08.799 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:50:08.799765 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:50:18.864 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:50:18.864591 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:50:18.902 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:50:18.903237 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:50:29.048 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:50:29.048702 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:50:29.061 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:50:29.062249 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:50:39.134 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:50:39.135504 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:50:39.151 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:50:39.151758 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:50:49.274 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:50:49.275447 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:50:49.285 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:50:49.286436 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:50:59.408 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:50:59.409145 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:50:59.423 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:50:59.423926 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:51:09.534 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:51:09.534998 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:51:09.547 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:51:09.548769 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:51:19.623 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:51:19.624165 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:51:19.663 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:51:19.664313 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:51:29.771 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:51:29.772251 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:51:29.785 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:51:29.786237 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:51:39.890 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:51:39.890725 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:51:39.899 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:51:39.900418 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:51:50.060 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:51:50.061259 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:51:50.071 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:51:50.072499 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:52:00.176 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:52:00.176701 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:52:00.186 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:52:00.187664 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:52:10.307 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:52:10.308516 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:52:10.342 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:52:10.343373 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:52:20.443 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:52:20.443640 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:52:20.455 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:52:20.456171 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:52:30.518 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:52:30.518921 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:52:30.578 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:52:30.579420 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:52:40.675 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:52:40.676470 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:52:40.687 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:52:40.688887 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:52:50.801 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:52:50.801623 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:52:50.816 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:52:50.816939 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:53:00.915 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:53:00.916438 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:53:00.928 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:53:00.929569 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:53:11.066 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:53:11.066643 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:53:11.093 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:53:11.094256 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:53:21.226 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:53:21.226930 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:53:21.239 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:53:21.240329 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:53:31.389 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:53:31.389643 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:53:31.398 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:53:31.399859 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:53:41.464 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:53:41.465454 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:53:41.477 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:53:41.478768 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:53:51.541 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:53:51.541839 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:53:51.607 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:53:51.608496 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:54:01.683 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:54:01.684552 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:54:01.695 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:54:01.696336 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:54:11.806 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:54:11.807072 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:54:11.819 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:54:11.820415 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:54:21.926 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:54:21.926700 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:54:21.939 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:54:21.940013 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:54:32.040 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:54:32.041494 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:54:32.052 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:54:32.053967 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:54:42.118 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:54:42.118930 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:54:42.130 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:54:42.130538 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:54:52.198 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:54:52.199670 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:54:52.208 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:54:52.209991 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:55:02.293 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:55:02.294286 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:55:02.305 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:55:02.306740 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:55:12.372 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:55:12.373621 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:55:12.386 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:55:12.387719 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:55:22.489 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:55:22.490354 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:55:22.500 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:55:22.501144 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:55:32.563 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:55:32.562206 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:55:32.572 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:55:32.574000 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:55:42.693 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:55:42.694555 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:55:42.704 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:55:42.705474 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:55:52.811 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:55:52.811819 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:55:52.823 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:55:52.824273 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:56:02.958 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:56:02.958922 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:56:02.973 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:56:02.974110 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:56:13.086 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:56:13.086784 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:56:13.099 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:56:13.100369 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:56:23.159 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:56:23.160155 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:56:23.170 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:56:23.171631 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:56:33.238 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:56:33.238756 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:56:33.249 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:56:33.250414 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:56:43.324 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:56:43.325029 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:56:43.387 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:56:43.388306 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:56:53.473 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:56:53.473896 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:56:53.481 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:56:53.483031 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:57:03.553 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:57:03.554562 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:57:03.566 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:57:03.567950 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:57:13.656 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:57:13.657606 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:57:13.669 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:57:13.669821 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:57:23.779 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:57:23.780268 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:57:23.790 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:57:23.791704 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:57:33.863 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:57:33.863982 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:57:33.877 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:57:33.878237 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:57:43.949 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:57:43.950545 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:57:43.960 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:57:43.961435 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:57:54.036 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:57:54.037423 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:57:54.060 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:57:54.056957 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:58:04.186 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:58:04.187271 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:58:04.225 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:58:04.226002 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:58:14.322 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:58:14.322756 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:58:14.333 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:58:14.334471 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:58:24.424 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:58:24.425108 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:58:24.435 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:58:24.436760 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:58:34.523 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:58:34.524149 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:58:34.682 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:58:34.537304 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:58:44.601 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:58:44.602154 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:58:44.611 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:58:44.612655 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:58:54.678 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:58:54.678900 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:58:54.692 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:58:54.692959 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:59:04.790 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:59:04.790584 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:59:04.802 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:59:04.803523 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:59:14.865 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:59:14.864760 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:59:14.878 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:59:14.879385 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:59:24.954 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:59:24.954904 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:59:24.966 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:59:24.967794 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:59:35.065 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:59:35.066110 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:59:35.078 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:59:35.079652 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:59:45.160 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:59:45.161235 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:59:45.175 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:59:45.176684 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:59:55.301 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:59:55.301491 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:59:55.310 INFO:tasks.rook.operator.smithi077.stdout:2022-03-08 23:59:55.311016 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:00:05.419 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:00:05.419609 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:00:05.429 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:00:05.430224 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:00:15.527 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:00:15.528368 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:00:15.539 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:00:15.540209 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:00:25.606 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:00:25.607513 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:00:25.616 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:00:25.617104 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:00:35.721 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:00:35.722543 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:00:35.734 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:00:35.735141 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:00:45.832 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:00:45.832760 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:00:45.845 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:00:45.846357 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:00:55.970 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:00:55.970887 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:00:55.981 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:00:55.982579 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:01:06.162 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:01:06.162807 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:01:06.171 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:01:06.172819 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:01:16.281 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:01:16.281431 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:01:16.299 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:01:16.295639 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:01:26.360 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:01:26.361338 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:01:26.371 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:01:26.372558 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:01:36.529 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:01:36.529923 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:01:36.543 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:01:36.544568 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:01:46.602 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:01:46.603442 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:01:46.614 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:01:46.615831 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:01:56.800 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:01:56.793889 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:01:56.806 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:01:56.807521 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:02:06.882 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:02:06.883028 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:02:06.895 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:02:06.895756 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:02:16.994 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:02:16.995109 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:02:17.008 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:02:17.009157 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:02:27.076 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:02:27.076990 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:02:27.142 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:02:27.142885 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:02:37.209 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:02:37.209511 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:02:37.220 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:02:37.221039 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:02:47.317 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:02:47.317845 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:02:47.336 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:02:47.333917 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:02:57.397 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:02:57.397707 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:02:57.406 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:02:57.407367 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:03:07.509 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:03:07.510565 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:03:07.519 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:03:07.520034 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:03:17.632 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:03:17.632736 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:03:17.646 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:03:17.646763 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:03:27.761 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:03:27.762087 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:03:27.772 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:03:27.773644 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:03:37.840 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:03:37.841585 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:03:37.853 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:03:37.854061 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:03:47.964 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:03:47.965493 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:03:47.976 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:03:47.977728 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:03:58.087 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:03:58.088318 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:03:58.099 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:03:58.100657 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:04:08.217 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:04:08.217762 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:04:08.233 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:04:08.234121 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:04:18.346 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:04:18.346590 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:04:18.358 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:04:18.359198 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:04:28.434 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:04:28.435554 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:04:28.447 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:04:28.448662 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:04:38.599 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:04:38.599666 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:04:38.619 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:04:38.611687 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:04:48.741 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:04:48.741575 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:04:48.754 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:04:48.754630 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:04:58.871 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:04:58.871775 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:04:58.880 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:04:58.881916 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:05:08.982 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:05:08.982552 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:05:08.996 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:05:08.996942 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:05:19.086 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:05:19.086516 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:05:19.163 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:05:19.163803 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:05:29.263 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:05:29.263634 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:05:29.276 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:05:29.277120 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:05:39.370 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:05:39.370576 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:05:39.381 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:05:39.382464 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:05:49.510 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:05:49.511161 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:05:49.525 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:05:49.526047 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:05:59.630 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:05:59.630918 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:05:59.647 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:05:59.646600 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:06:09.725 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:06:09.726298 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:06:09.738 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:06:09.739373 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:06:19.819 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:06:19.819479 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:06:19.864 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:06:19.864625 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:06:29.951 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:06:29.952259 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:06:29.966 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:06:29.967626 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:06:36.857 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:06:36.854523 I | op-mon: parsing mon endpoints: a=10.110.141.123:6789,b=10.98.41.230:6789,c=10.109.47.142:6789 2022-03-09T00:06:37.491 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:06:37.491799 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-09T00:06:37.479+0000 7f63dad2d380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-09T00:06:37.491 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09T00:06:37.479+0000 7f63dad2d380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-09T00:06:37.492 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09T00:06:37.479+0000 7f63dad2d380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-09T00:06:37.492 INFO:tasks.rook.operator.smithi077.stdout:couldn't init storage provider. : exit status 5 2022-03-09T00:06:40.063 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:06:40.064119 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:06:40.077 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:06:40.077972 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:06:50.154 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:06:50.155401 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:06:50.165 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:06:50.166307 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:07:00.263 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:07:00.263976 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:07:00.274 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:07:00.274995 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:07:10.382 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:07:10.382542 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:07:10.392 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:07:10.393184 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:07:20.508 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:07:20.509279 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:07:20.523 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:07:20.524682 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:07:30.590 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:07:30.591601 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:07:30.603 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:07:30.604197 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:07:40.683 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:07:40.684515 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:07:40.753 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:07:40.753963 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:07:50.901 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:07:50.902152 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:07:50.913 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:07:50.914487 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:08:01.027 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:08:01.028056 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:08:01.038 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:08:01.039387 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:08:11.156 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:08:11.156982 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:08:11.168 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:08:11.169967 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:08:21.285 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:08:21.286382 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:08:21.296 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:08:21.297206 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:08:31.361 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:08:31.362608 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:08:31.439 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:08:31.440816 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:08:41.497 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:08:41.496475 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:08:41.508 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:08:41.509871 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:08:51.615 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:08:51.615713 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:08:51.626 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:08:51.627458 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:09:01.695 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:09:01.695667 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:09:01.708 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:09:01.709261 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:09:11.844 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:09:11.845229 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:09:11.856 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:09:11.857693 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:09:21.931 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:09:21.931876 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:09:21.992 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:09:21.993085 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:09:32.101 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:09:32.101720 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:09:32.114 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:09:32.115649 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:09:42.207 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:09:42.207991 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:09:42.220 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:09:42.221669 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:09:52.316 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:09:52.317631 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:09:52.331 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:09:52.332659 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:10:02.423 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:10:02.424398 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:10:02.440 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:10:02.436501 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:10:12.510 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:10:12.510623 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:10:12.522 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:10:12.523611 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:10:22.635 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:10:22.636298 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:10:22.649 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:10:22.650122 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:10:32.711 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:10:32.711675 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:10:32.724 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:10:32.725510 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:10:42.781 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:10:42.778990 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:10:42.792 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:10:42.793354 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:10:52.962 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:10:52.962774 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:10:52.974 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:10:52.975727 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:11:03.084 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:11:03.085184 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:11:03.096 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:11:03.097548 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:11:13.186 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:11:13.186989 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:11:13.200 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:11:13.200996 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:11:23.271 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:11:23.272318 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:11:23.281 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:11:23.282758 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:11:33.398 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:11:33.399043 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:11:33.409 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:11:33.410487 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:11:43.475 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:11:43.476368 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:11:43.788 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:11:43.487573 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:11:53.544 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:11:53.540983 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:11:53.552 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:11:53.553833 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:12:03.655 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:12:03.656568 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:12:03.669 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:12:03.669960 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:12:13.737 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:12:13.738664 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:12:13.752 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:12:13.753824 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:12:23.828 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:12:23.829447 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:12:23.839 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:12:23.840965 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:12:33.911 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:12:33.912642 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:12:33.961 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:12:33.961866 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:12:44.035 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:12:44.036913 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:12:44.049 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:12:44.050445 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:12:54.156 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:12:54.157471 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:12:54.167 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:12:54.168280 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:13:04.297 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:13:04.298276 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:13:04.311 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:13:04.312325 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:13:14.369 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:13:14.370425 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:13:14.381 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:13:14.382362 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:13:24.506 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:13:24.507542 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:13:24.516 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:13:24.518136 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:13:34.589 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:13:34.589929 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:13:34.602 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:13:34.603047 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:13:44.730 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:13:44.730723 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:13:44.742 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:13:44.743196 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:13:54.806 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:13:54.807587 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:13:54.819 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:13:54.820247 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:14:04.937 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:14:04.937878 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:14:04.950 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:14:04.951505 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:14:15.092 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:14:15.093044 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:14:15.103 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:14:15.104523 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:14:25.218 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:14:25.219518 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:14:25.233 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:14:25.234636 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:14:35.347 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:14:35.348390 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:14:35.360 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:14:35.361064 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:14:45.639 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:14:45.639767 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:14:45.649 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:14:45.650626 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:14:55.772 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:14:55.772855 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:14:55.784 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:14:55.785229 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:15:05.852 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:15:05.852961 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:15:05.862 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:15:05.863666 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:15:15.969 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:15:15.969716 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:15:15.980 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:15:15.980981 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:15:26.088 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:15:26.089245 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:15:26.130 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:15:26.131706 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:15:36.207 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:15:36.207867 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:15:36.220 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:15:36.221765 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:15:46.371 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:15:46.371889 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:15:46.382 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:15:46.382928 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:15:56.463 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:15:56.464521 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:15:56.549 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:15:56.548355 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:16:06.719 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:16:06.719797 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:16:06.729 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:16:06.730260 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:16:16.794 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:16:16.795098 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:16:16.807 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:16:16.807736 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:16:26.874 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:16:26.874990 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:16:26.934 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:16:26.930799 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:16:37.027 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:16:37.027905 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:16:37.041 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:16:37.041838 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:16:47.164 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:16:47.164336 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:16:47.177 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:16:47.178120 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:16:57.243 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:16:57.243616 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:16:57.257 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:16:57.258128 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:17:07.326 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:17:07.326931 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:17:07.390 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:17:07.391400 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:17:17.472 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:17:17.473155 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:17:17.484 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:17:17.485112 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:17:27.568 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:17:27.568955 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:17:27.582 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:17:27.582919 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:17:37.699 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:17:37.699920 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:17:37.712 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:17:37.712914 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:17:47.827 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:17:47.827229 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:17:47.837 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:17:47.838010 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:17:57.919 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:17:57.919653 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:17:57.931 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:17:57.931800 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:18:08.053 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:18:08.053368 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:18:08.063 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:18:08.064243 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:18:18.149 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:18:18.149818 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:18:18.161 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:18:18.162423 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:18:28.283 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:18:28.283826 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:18:28.335 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:18:28.335856 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:18:38.533 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:18:38.533907 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:18:38.546 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:18:38.546833 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:18:48.602 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:18:48.603193 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:18:48.627 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:18:48.616710 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:18:58.700 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:18:58.700513 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:18:58.711 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:18:58.711776 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:19:08.812 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:19:08.812866 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:19:08.826 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:19:08.826967 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:19:18.921 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:19:18.922022 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:19:18.936 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:19:18.937085 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:19:29.038 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:19:29.038587 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:19:29.051 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:19:29.051940 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:19:39.123 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:19:39.123484 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:19:39.136 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:19:39.137252 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:19:49.256 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:19:49.256436 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:19:49.268 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:19:49.268659 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:19:59.383 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:19:59.383812 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:19:59.397 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:19:59.398396 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:20:09.484 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:20:09.485220 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:20:09.500 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:20:09.501178 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:20:19.578 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:20:19.577894 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:20:19.591 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:20:19.591873 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:20:29.729 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:20:29.729110 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:20:29.741 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:20:29.741485 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:20:39.808 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:20:39.808755 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:20:39.822 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:20:39.822612 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:20:49.940 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:20:49.940087 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:20:49.950 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:20:49.950856 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:21:00.052 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:21:00.052760 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:21:00.065 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:21:00.065586 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:21:10.153 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:21:10.153244 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:21:10.164 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:21:10.164940 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:21:20.233 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:21:20.233594 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:21:20.248 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:21:20.248369 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:21:30.321 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:21:30.321208 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:21:30.333 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:21:30.333668 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:21:40.414 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:21:40.414554 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:21:40.430 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:21:40.430308 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:21:50.491 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:21:50.491144 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:21:50.501 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:21:50.502072 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:22:00.624 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:22:00.624312 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:22:00.637 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:22:00.637604 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:22:10.782 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:22:10.782685 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:22:10.792 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:22:10.793052 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:22:20.933 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:22:20.933464 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:22:20.944 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:22:20.944436 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:22:31.040 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:22:31.040478 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:22:31.054 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:22:31.055034 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:22:41.139 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:22:41.139718 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:22:41.150 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:22:41.150358 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:22:51.265 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:22:51.265589 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:22:51.276 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:22:51.277122 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:23:01.380 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:23:01.380335 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:23:01.393 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:23:01.393970 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:23:11.493 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:23:11.493375 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:23:11.505 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:23:11.505953 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:23:17.500 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:23:17.500569 I | op-mon: parsing mon endpoints: a=10.110.141.123:6789,b=10.98.41.230:6789,c=10.109.47.142:6789 2022-03-09T00:23:18.148 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:23:18.147757 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-09T00:23:18.137+0000 7f0fdf8fe380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-09T00:23:18.148 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09T00:23:18.137+0000 7f0fdf8fe380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-09T00:23:18.148 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09T00:23:18.137+0000 7f0fdf8fe380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-09T00:23:18.149 INFO:tasks.rook.operator.smithi077.stdout:couldn't init storage provider. : exit status 5 2022-03-09T00:23:21.606 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:23:21.606623 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:23:21.617 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:23:21.618215 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:23:31.690 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:23:31.690605 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:23:31.703 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:23:31.703688 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:23:41.834 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:23:41.834399 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:23:41.844 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:23:41.844928 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:23:51.945 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:23:51.945490 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:23:51.958 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:23:51.958315 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:24:02.057 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:24:02.056941 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:24:02.069 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:24:02.069981 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:24:12.155 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:24:12.155068 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:24:12.163 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:24:12.163939 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:24:22.226 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:24:22.226261 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:24:22.280 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:24:22.279858 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:24:32.409 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:24:32.409393 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:24:32.431 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:24:32.431856 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:24:42.528 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:24:42.527879 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:24:42.543 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:24:42.543163 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:24:52.620 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:24:52.620546 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:24:52.632 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:24:52.632820 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:25:02.709 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:25:02.708959 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:25:02.721 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:25:02.721691 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:25:12.799 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:25:12.799672 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:25:12.812 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:25:12.812457 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:25:22.917 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:25:22.917239 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:25:22.932 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:25:22.932242 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:25:33.028 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:25:33.028660 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:25:33.065 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:25:33.041790 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:25:43.150 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:25:43.150433 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:25:43.161 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:25:43.161997 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:25:53.263 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:25:53.263504 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:25:53.274 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:25:53.274547 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:26:03.358 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:26:03.358058 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:26:03.368 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:26:03.368416 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:26:13.472 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:26:13.472327 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:26:13.487 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:26:13.487852 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:26:23.605 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:26:23.605600 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:26:23.618 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:26:23.618861 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:26:33.688 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:26:33.688569 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:26:33.703 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:26:33.703890 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:26:43.818 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:26:43.818619 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:26:43.831 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:26:43.831815 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:26:53.918 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:26:53.918372 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:26:53.932 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:26:53.932141 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:27:04.000 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:27:03.999954 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:27:04.010 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:27:04.011250 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:27:14.091 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:27:14.091505 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:27:14.163 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:27:14.163825 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:27:24.234 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:27:24.234474 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:27:24.247 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:27:24.248209 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:27:34.366 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:27:34.366583 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:27:34.376 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:27:34.376417 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:27:44.515 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:27:44.514868 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:27:44.524 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:27:44.524954 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:27:54.696 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:27:54.696447 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:27:54.732 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:27:54.732448 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:28:04.833 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:28:04.833318 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:28:04.845 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:28:04.846109 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:28:14.947 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:28:14.947180 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:28:14.961 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:28:14.961703 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:28:25.039 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:28:25.039066 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:28:25.094 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:28:25.094544 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:28:35.218 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:28:35.218259 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:28:35.231 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:28:35.231417 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:28:45.313 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:28:45.313623 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:28:45.326 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:28:45.326417 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:28:55.442 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:28:55.442893 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:28:55.454 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:28:55.454052 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:29:05.511 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:29:05.507818 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:29:05.517 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:29:05.516555 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:29:15.603 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:29:15.604029 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:29:15.612 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:29:15.613095 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:29:25.685 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:29:25.684948 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:29:25.695 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:29:25.696274 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:29:35.755 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:29:35.755072 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:29:35.765 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:29:35.765882 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:29:45.892 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:29:45.892280 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:29:45.905 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:29:45.905153 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:29:56.018 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:29:56.018683 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:29:56.031 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:29:56.031906 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:30:06.112 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:30:06.112777 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:30:06.125 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:30:06.125399 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:30:16.213 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:30:16.213594 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:30:16.226 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:30:16.226633 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:30:26.346 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:30:26.340652 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:30:26.351 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:30:26.352160 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:30:36.465 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:30:36.465262 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:30:36.479 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:30:36.479897 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:30:46.540 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:30:46.540271 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:30:46.552 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:30:46.552386 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:30:56.715 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:30:56.715185 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:30:56.725 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:30:56.726138 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:31:06.844 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:31:06.843694 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:31:06.858 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:31:06.858715 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:31:16.986 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:31:16.942984 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:31:16.986 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:31:16.955847 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:31:27.050 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:31:27.050516 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:31:27.063 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:31:27.063780 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:31:37.126 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:31:37.126930 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:31:37.174 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:31:37.175026 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:31:47.241 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:31:47.241770 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:31:47.307 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:31:47.307861 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:31:57.381 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:31:57.381893 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:31:57.446 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:31:57.446854 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:32:07.507 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:32:07.507414 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:32:07.520 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:32:07.519972 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:32:17.592 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:32:17.591999 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:32:17.608 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:32:17.609233 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:32:27.693 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:32:27.693538 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:32:27.742 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:32:27.742108 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:32:37.809 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:32:37.809110 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:32:37.821 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:32:37.820701 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:32:47.892 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:32:47.892510 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:32:47.941 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:32:47.941282 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:32:58.071 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:32:58.071058 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:32:58.083 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:32:58.084083 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:33:08.143 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:33:08.142928 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:33:08.153 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:33:08.154113 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:33:18.226 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:33:18.226267 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:33:18.238 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:33:18.239042 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:33:28.320 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:33:28.320758 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:33:28.333 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:33:28.333480 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:33:38.443 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:33:38.443865 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:33:38.456 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:33:38.456463 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:33:48.565 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:33:48.565537 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:33:48.579 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:33:48.579561 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:33:58.659 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:33:58.659698 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:33:58.671 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:33:58.672036 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:34:08.765 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:34:08.765862 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:34:08.777 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:34:08.777603 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:34:18.896 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:34:18.896183 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:34:18.910 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:34:18.911008 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:34:29.022 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:34:29.022657 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:34:29.035 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:34:29.035098 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:34:39.107 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:34:39.107366 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:34:39.118 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:34:39.118396 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:34:49.195 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:34:49.195125 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:34:49.206 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:34:49.207162 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:34:59.343 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:34:59.343656 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:34:59.360 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:34:59.360490 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:35:09.435 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:35:09.435134 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:35:09.447 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:35:09.447812 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:35:19.525 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:35:19.525271 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:35:19.538 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:35:19.539031 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:35:29.599 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:35:29.599558 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:35:29.613 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:35:29.611843 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:35:39.718 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:35:39.718540 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:35:39.729 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:35:39.730049 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:35:49.802 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:35:49.801918 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:35:49.814 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:35:49.814726 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:35:59.927 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:35:59.927025 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:35:59.940 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:35:59.940704 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:36:10.016 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:36:10.016553 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:36:10.031 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:36:10.031659 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:36:20.144 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:36:20.144502 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:36:20.156 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:36:20.156560 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:36:30.260 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:36:30.260712 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:36:30.273 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:36:30.273749 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:36:40.352 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:36:40.351821 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:36:40.367 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:36:40.367672 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:36:50.430 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:36:50.430021 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:36:50.442 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:36:50.442632 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:37:00.549 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:37:00.549739 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:37:00.561 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:37:00.562097 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:37:10.675 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:37:10.674901 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:37:10.693 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:37:10.689367 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:37:20.813 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:37:20.813654 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:37:20.825 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:37:20.825765 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:37:30.892 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:37:30.892018 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:37:30.952 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:37:30.952132 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:37:41.063 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:37:41.063176 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:37:41.075 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:37:41.075198 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:37:51.159 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:37:51.159660 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:37:51.178 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:37:51.173252 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:38:01.232 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:38:01.231178 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:38:01.243 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:38:01.244119 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:38:11.307 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:38:11.307537 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:38:11.357 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:38:11.357706 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:38:21.478 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:38:21.478206 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:38:21.490 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:38:21.490278 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:38:31.579 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:38:31.578734 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:38:31.593 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:38:31.593371 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:38:41.763 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:38:41.763176 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:38:41.772 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:38:41.772900 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:38:51.832 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:38:51.832393 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:38:51.845 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:38:51.845279 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:39:01.947 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:39:01.947376 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:39:01.959 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:39:01.959680 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:39:12.050 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:39:12.050332 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:39:12.062 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:39:12.062725 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:39:22.189 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:39:22.188857 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:39:22.201 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:39:22.201693 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:39:32.287 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:39:32.286987 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:39:32.302 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:39:32.302487 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:39:42.373 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:39:42.372932 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:39:42.421 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:39:42.421099 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:39:52.486 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:39:52.486114 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:39:52.497 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:39:52.497295 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:39:58.157 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:39:58.157488 I | op-mon: parsing mon endpoints: a=10.110.141.123:6789,b=10.98.41.230:6789,c=10.109.47.142:6789 2022-03-09T00:39:58.811 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:39:58.810709 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-09T00:39:58.800+0000 7f65f89d3380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-09T00:39:58.812 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09T00:39:58.800+0000 7f65f89d3380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-09T00:39:58.812 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09T00:39:58.800+0000 7f65f89d3380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-09T00:39:58.813 INFO:tasks.rook.operator.smithi077.stdout:couldn't init storage provider. : exit status 5 2022-03-09T00:40:02.573 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:40:02.573656 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:40:02.586 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:40:02.586085 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:40:12.727 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:40:12.727747 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:40:12.761 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:40:12.761881 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:40:22.864 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:40:22.863545 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:40:22.883 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:40:22.883750 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:40:33.052 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:40:33.052013 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:40:33.068 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:40:33.068416 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:40:43.142 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:40:43.142331 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:40:43.155 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:40:43.154904 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:40:53.230 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:40:53.229900 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:40:53.243 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:40:53.243265 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:41:03.325 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:41:03.325269 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:41:03.338 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:41:03.339043 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:41:13.455 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:41:13.455755 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:41:13.468 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:41:13.468077 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:41:23.609 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:41:23.608584 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:41:23.623 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:41:23.623939 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:41:33.737 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:41:33.737687 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:41:33.756 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:41:33.751870 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:41:43.865 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:41:43.864739 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:41:43.877 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:41:43.877156 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:41:53.942 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:41:53.942025 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:41:53.954 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:41:53.954318 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:42:04.052 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:42:04.051922 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:42:04.065 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:42:04.065743 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:42:14.171 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:42:14.171281 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:42:14.185 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:42:14.185361 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:42:24.254 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:42:24.254119 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:42:24.372 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:42:24.371870 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:42:34.431 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:42:34.431638 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:42:34.443 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:42:34.444012 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:42:44.539 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:42:44.538958 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:42:44.552 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:42:44.552193 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:42:54.639 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:42:54.639199 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:42:54.653 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:42:54.653506 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:43:04.724 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:43:04.724768 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:43:04.738 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:43:04.738551 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:43:14.818 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:43:14.818479 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:43:14.833 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:43:14.833311 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:43:24.898 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:43:24.897610 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:43:24.911 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:43:24.911032 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:43:35.015 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:43:35.015100 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:43:35.026 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:43:35.026985 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:43:45.157 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:43:45.157762 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:43:45.168 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:43:45.168653 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:43:55.250 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:43:55.250396 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:43:55.263 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:43:55.263916 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:44:05.329 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:44:05.329672 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:44:05.386 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:44:05.386846 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:44:15.478 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:44:15.478111 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:44:15.490 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:44:15.490934 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:44:25.554 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:44:25.553642 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:44:25.633 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:44:25.633719 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:44:35.701 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:44:35.701288 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:44:35.713 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:44:35.713828 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:44:45.820 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:44:45.819855 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:44:45.851 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:44:45.851473 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:44:55.982 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:44:55.981933 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:44:56.069 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:44:55.993580 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:45:06.094 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:45:06.093624 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:45:06.106 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:45:06.106568 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:45:16.179 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:45:16.179581 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:45:16.192 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:45:16.193065 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:45:26.270 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:45:26.270383 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:45:26.321 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:45:26.321216 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:45:36.394 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:45:36.394110 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:45:36.407 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:45:36.407459 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:45:46.495 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:45:46.495736 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:45:46.508 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:45:46.508143 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:45:56.604 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:45:56.604016 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:45:56.616 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:45:56.616253 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:46:06.715 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:46:06.714906 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:46:06.726 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:46:06.726484 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:46:16.823 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:46:16.823095 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:46:16.837 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:46:16.837793 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:46:26.902 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:46:26.902233 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:46:26.916 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:46:26.915282 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:46:37.042 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:46:37.042740 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:46:37.054 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:46:37.054707 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:46:47.126 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:46:47.125540 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:46:47.136 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:46:47.136128 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:46:57.225 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:46:57.225651 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:46:57.234 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:46:57.234252 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:47:07.332 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:47:07.331642 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:47:07.343 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:47:07.343952 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:47:17.417 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:47:17.416702 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:47:17.431 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:47:17.428845 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:47:27.558 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:47:27.558104 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:47:27.571 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:47:27.571398 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:47:37.638 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:47:37.638133 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:47:37.650 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:47:37.649905 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:47:47.778 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:47:47.778215 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:47:47.788 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:47:47.788255 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:47:57.891 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:47:57.891651 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:47:57.903 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:47:57.903373 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:48:07.976 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:48:07.976379 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:48:07.988 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:48:07.988339 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:48:18.061 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:48:18.061637 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:48:18.075 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:48:18.075160 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:48:28.203 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:48:28.203621 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:48:28.215 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:48:28.215268 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:48:38.305 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:48:38.304868 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:48:38.318 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:48:38.318241 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:48:48.494 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:48:48.494197 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:48:48.504 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:48:48.504752 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:48:58.666 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:48:58.666548 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:48:58.678 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:48:58.678611 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:49:08.762 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:49:08.761951 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:49:08.775 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:49:08.775408 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:49:18.844 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:49:18.844177 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:49:18.885 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:49:18.885424 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:49:28.975 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:49:28.975029 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:49:28.987 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:49:28.987367 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:49:39.065 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:49:39.065149 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:49:39.128 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:49:39.128109 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:49:49.240 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:49:49.240694 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:49:49.255 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:49:49.256132 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:49:59.327 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:49:59.327372 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:49:59.341 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:49:59.341101 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:50:09.411 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:50:09.410860 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:50:09.422 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:50:09.422235 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:50:19.497 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:50:19.497164 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:50:19.508 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:50:19.508319 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:50:29.622 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:50:29.622177 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:50:29.639 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:50:29.640040 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:50:39.709 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:50:39.709041 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:50:39.721 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:50:39.721699 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:50:49.837 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:50:49.837705 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:50:49.849 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:50:49.849377 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:50:59.970 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:50:59.969814 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:50:59.982 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:50:59.982276 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:51:10.044 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:51:10.043926 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:51:10.057 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:51:10.057057 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:51:20.174 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:51:20.174143 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:51:20.188 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:51:20.188499 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:51:30.292 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:51:30.292185 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:51:30.303 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:51:30.304178 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:51:40.413 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:51:40.412908 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:51:40.425 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:51:40.425157 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:51:50.563 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:51:50.563055 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:51:50.574 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:51:50.574370 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:52:00.720 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:52:00.720781 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:52:00.730 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:52:00.730508 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:52:10.867 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:52:10.867266 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:52:10.881 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:52:10.881446 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:52:20.949 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:52:20.949496 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:52:20.959 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:52:20.959732 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:52:31.042 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:52:31.042550 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:52:31.053 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:52:31.053657 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:52:41.133 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:52:41.133453 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:52:41.198 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:52:41.198708 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:52:51.263 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:52:51.263623 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:52:51.276 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:52:51.276768 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:53:01.345 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:53:01.345269 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:53:01.356 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:53:01.357040 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:53:11.463 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:53:11.463701 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:53:11.474 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:53:11.474996 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:53:21.583 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:53:21.582139 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:53:21.600 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:53:21.600556 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:53:31.730 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:53:31.729610 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:53:31.742 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:53:31.742560 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:53:41.814 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:53:41.814709 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:53:41.830 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:53:41.830305 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:53:51.957 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:53:51.957466 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:53:51.971 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:53:51.971921 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:54:02.129 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:54:02.129112 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:54:02.143 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:54:02.143494 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:54:12.263 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:54:12.263303 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:54:12.275 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:54:12.275840 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:54:22.345 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:54:22.345974 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:54:22.460 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:54:22.460318 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:54:32.522 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:54:32.522712 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:54:32.537 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:54:32.537525 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:54:42.649 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:54:42.649602 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:54:42.661 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:54:42.661185 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:54:52.746 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:54:52.746013 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:54:52.758 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:54:52.758861 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:55:02.839 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:55:02.839180 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:55:02.852 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:55:02.852943 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:55:12.924 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:55:12.924655 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:55:12.936 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:55:12.936385 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:55:23.050 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:55:23.050459 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:55:23.064 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:55:23.064875 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:55:33.178 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:55:33.178134 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:55:33.190 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:55:33.190959 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:55:43.359 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:55:43.359709 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:55:43.372 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:55:43.373020 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:55:53.477 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:55:53.477256 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:55:53.495 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:55:53.496069 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:56:03.558 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:56:03.558254 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:56:03.569 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:56:03.569969 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:56:13.624 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:56:13.624303 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:56:13.638 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:56:13.638840 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:56:23.710 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:56:23.710671 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:56:23.723 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:56:23.723484 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:56:33.791 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:56:33.791791 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:56:33.805 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:56:33.805692 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:56:38.821 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:56:38.821360 I | op-mon: parsing mon endpoints: a=10.110.141.123:6789,b=10.98.41.230:6789,c=10.109.47.142:6789 2022-03-09T00:56:39.470 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:56:39.468550 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-09T00:56:39.455+0000 7f6b09c85380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-09T00:56:39.471 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09T00:56:39.455+0000 7f6b09c85380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-09T00:56:39.471 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09T00:56:39.455+0000 7f6b09c85380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-09T00:56:39.471 INFO:tasks.rook.operator.smithi077.stdout:couldn't init storage provider. : exit status 5 2022-03-09T00:56:43.868 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:56:43.868046 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:56:43.881 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:56:43.881737 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:56:53.989 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:56:53.989856 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:56:54.002 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:56:54.002789 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:57:04.139 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:57:04.139626 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:57:04.150 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:57:04.151196 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:57:14.221 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:57:14.221663 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:57:14.237 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:57:14.237694 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:57:24.352 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:57:24.352123 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:57:24.365 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:57:24.365758 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:57:34.478 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:57:34.478732 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:57:34.490 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:57:34.490660 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:57:44.546 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:57:44.544992 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:57:44.557 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:57:44.557710 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:57:54.632 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:57:54.632661 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:57:54.667 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:57:54.667198 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:58:04.765 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:58:04.765690 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:58:04.787 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:58:04.787757 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:58:14.899 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:58:14.899208 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:58:14.912 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:58:14.913078 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:58:24.976 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:58:24.976609 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:58:25.036 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:58:25.036464 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:58:35.154 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:58:35.154905 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:58:35.168 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:58:35.167578 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:58:45.257 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:58:45.257723 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:58:45.271 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:58:45.271564 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:58:55.338 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:58:55.337934 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:58:55.385 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:58:55.385651 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:59:05.456 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:59:05.457021 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:59:05.469 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:59:05.469810 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:59:15.561 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:59:15.561613 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:59:15.574 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:59:15.574711 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:59:25.649 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:59:25.649572 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:59:25.659 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:59:25.660085 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:59:35.818 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:59:35.818595 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:59:35.829 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:59:35.830149 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:59:45.905 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:59:45.905132 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:59:45.918 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:59:45.918302 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:59:56.024 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:59:56.024896 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:59:56.037 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 00:59:56.037809 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:00:06.159 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:00:06.159694 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:00:06.178 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:00:06.178134 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:00:16.248 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:00:16.248928 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:00:16.258 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:00:16.258915 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:00:26.351 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:00:26.351252 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:00:26.363 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:00:26.363504 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:00:36.422 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:00:36.422754 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:00:36.436 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:00:36.436500 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:00:46.534 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:00:46.534767 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:00:46.548 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:00:46.549048 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:00:56.621 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:00:56.621305 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:00:56.691 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:00:56.687511 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:01:06.793 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:01:06.793750 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:01:06.805 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:01:06.806329 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:01:16.958 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:01:16.958619 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:01:16.972 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:01:16.972714 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:01:27.086 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:01:27.086428 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:01:27.097 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:01:27.098131 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:01:37.213 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:01:37.213628 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:01:37.225 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:01:37.225029 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:01:47.287 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:01:47.287499 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:01:47.300 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:01:47.300985 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:01:57.379 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:01:57.379678 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:01:57.412 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:01:57.413055 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:02:07.526 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:02:07.526760 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:02:07.540 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:02:07.540620 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:02:17.609 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:02:17.609829 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:02:17.618 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:02:17.618752 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:02:27.715 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:02:27.715482 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:02:27.728 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:02:27.728733 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:02:37.832 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:02:37.832855 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:02:37.846 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:02:37.846987 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:02:47.955 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:02:47.955332 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:02:47.966 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:02:47.966497 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:02:58.084 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:02:58.084338 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:02:58.093 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:02:58.093799 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:03:08.199 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:03:08.199535 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:03:08.213 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:03:08.213619 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:03:18.334 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:03:18.334123 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:03:18.349 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:03:18.349131 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:03:28.458 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:03:28.458553 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:03:28.470 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:03:28.471182 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:03:38.564 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:03:38.564152 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:03:38.643 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:03:38.595870 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:03:48.764 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:03:48.763965 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:03:48.776 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:03:48.776091 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:03:58.872 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:03:58.872478 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:03:58.886 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:03:58.886582 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:04:08.996 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:04:08.996329 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:04:09.007 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:04:09.007683 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:04:19.142 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:04:19.142388 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:04:19.154 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:04:19.155143 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:04:29.264 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:04:29.264287 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:04:29.275 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:04:29.276123 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:04:39.347 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:04:39.347307 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:04:39.406 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:04:39.406590 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:04:49.512 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:04:49.512714 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:04:49.526 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:04:49.527198 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:04:59.654 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:04:59.654930 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:04:59.666 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:04:59.666935 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:05:09.847 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:05:09.847557 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:05:09.859 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:05:09.859672 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:05:19.964 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:05:19.964579 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:05:19.975 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:05:19.975960 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:05:30.043 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:05:30.044012 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:05:30.054 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:05:30.054894 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:05:40.177 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:05:40.177947 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:05:40.190 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:05:40.190570 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:05:50.267 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:05:50.266717 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:05:50.279 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:05:50.279660 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:06:00.397 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:06:00.398039 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:06:00.414 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:06:00.411571 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:06:10.524 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:06:10.524744 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:06:10.536 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:06:10.537276 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:06:20.626 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:06:20.626904 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:06:20.695 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:06:20.695553 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:06:30.814 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:06:30.814583 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:06:30.829 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:06:30.830452 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:06:40.942 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:06:40.942784 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:06:40.954 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:06:40.954442 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:06:51.048 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:06:51.047087 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:06:51.061 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:06:51.061224 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:07:01.143 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:07:01.144076 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:07:01.161 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:07:01.161914 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:07:11.308 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:07:11.309252 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:07:11.320 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:07:11.320704 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:07:21.400 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:07:21.400228 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:07:21.413 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:07:21.414355 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:07:31.504 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:07:31.504745 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:07:31.516 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:07:31.517299 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:07:41.577 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:07:41.572518 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:07:41.588 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:07:41.589241 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:07:51.651 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:07:51.651849 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:07:51.662 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:07:51.662876 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:08:01.728 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:08:01.728587 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:08:01.748 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:08:01.748644 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:08:11.817 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:08:11.818186 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:08:11.916 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:08:11.916334 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:08:22.036 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:08:22.036657 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:08:22.049 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:08:22.049740 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:08:32.116 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:08:32.116461 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:08:32.188 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:08:32.188737 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:08:42.298 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:08:42.298836 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:08:42.314 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:08:42.314980 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:08:52.418 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:08:52.418655 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:08:52.430 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:08:52.431090 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:09:02.503 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:09:02.503715 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:09:02.515 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:09:02.515572 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:09:12.623 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:09:12.624195 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:09:12.635 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:09:12.635906 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:09:22.700 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:09:22.700548 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:09:22.761 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:09:22.761771 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:09:32.873 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:09:32.873598 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:09:32.886 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:09:32.886971 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:09:42.967 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:09:42.967247 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:09:42.978 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:09:42.979229 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:09:53.091 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:09:53.091743 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:09:53.104 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:09:53.104948 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:10:03.244 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:10:03.244543 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:10:03.252 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:10:03.253362 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:10:13.395 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:10:13.395735 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:10:13.407 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:10:13.407696 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:10:23.502 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:10:23.502395 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:10:23.513 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:10:23.514317 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:10:33.619 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:10:33.619424 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:10:33.629 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:10:33.630649 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:10:43.750 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:10:43.750635 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:10:43.762 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:10:43.762816 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:10:53.857 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:10:53.857215 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:10:53.876 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:10:53.876536 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:11:03.985 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:11:03.986144 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:11:04.101 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:11:03.998713 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:11:14.141 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:11:14.142232 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:11:14.155 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:11:14.156747 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:11:24.275 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:11:24.275592 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:11:24.288 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:11:24.289037 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:11:34.402 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:11:34.402503 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:11:34.412 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:11:34.413324 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:11:44.476 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:11:44.477076 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:11:44.491 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:11:44.491919 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:11:54.546 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:11:54.545516 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:11:54.556 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:11:54.556840 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:12:04.620 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:12:04.620481 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:12:04.631 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:12:04.631812 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:12:14.697 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:12:14.697480 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:12:14.715 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:12:14.716075 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:12:24.873 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:12:24.873645 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:12:24.886 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:12:24.887104 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:12:34.979 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:12:34.979390 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:12:34.991 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:12:34.991800 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:12:45.050 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:12:45.050612 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:12:45.063 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:12:45.063774 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:12:55.226 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:12:55.226817 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:12:55.243 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:12:55.239699 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:13:05.351 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:13:05.352042 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:13:05.364 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:13:05.365249 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:13:15.468 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:13:15.468445 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:13:15.524 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:13:15.524535 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:13:19.476 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:13:19.477128 I | op-mon: parsing mon endpoints: a=10.110.141.123:6789,b=10.98.41.230:6789,c=10.109.47.142:6789 2022-03-09T01:13:20.132 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:13:20.132252 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-09T01:13:20.119+0000 7fcd4cdff380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-09T01:13:20.132 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09T01:13:20.119+0000 7fcd4cdff380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-09T01:13:20.132 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09T01:13:20.119+0000 7fcd4cdff380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-09T01:13:20.132 INFO:tasks.rook.operator.smithi077.stdout:couldn't init storage provider. : exit status 5 2022-03-09T01:13:25.658 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:13:25.659045 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:13:25.668 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:13:25.669546 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:13:35.819 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:13:35.819679 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:13:35.830 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:13:35.830752 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:13:45.917 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:13:45.918096 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:13:45.928 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:13:45.929252 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:13:56.029 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:13:56.029943 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:13:56.038 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:13:56.038978 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:14:06.181 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:14:06.182153 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:14:06.193 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:14:06.194403 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:14:16.266 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:14:16.267220 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:14:16.278 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:14:16.279244 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:14:26.338 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:14:26.338710 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:14:26.351 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:14:26.351671 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:14:36.470 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:14:36.470779 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:14:36.483 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:14:36.483918 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:14:46.548 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:14:46.549291 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:14:46.561 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:14:46.561704 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:14:56.657 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:14:56.657407 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:14:56.670 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:14:56.670918 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:15:06.779 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:15:06.779663 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:15:06.795 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:15:06.796349 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:15:16.939 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:15:16.939966 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:15:16.957 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:15:16.954321 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:15:27.056 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:15:27.056742 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:15:27.069 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:15:27.069658 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:15:37.203 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:15:37.203712 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:15:37.218 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:15:37.219210 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:15:47.325 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:15:47.325562 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:15:47.339 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:15:47.339597 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:15:57.435 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:15:57.435847 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:15:57.447 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:15:57.447669 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:16:07.561 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:16:07.560955 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:16:07.576 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:16:07.576662 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:16:17.645 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:16:17.645919 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:16:17.657 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:16:17.657798 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:16:27.766 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:16:27.766708 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:16:27.779 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:16:27.780319 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:16:37.842 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:16:37.842170 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:16:37.910 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:16:37.910862 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:16:47.985 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:16:47.986278 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:16:47.999 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:16:47.999834 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:16:58.092 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:16:58.092787 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:16:58.107 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:16:58.107630 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:17:08.240 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:17:08.240631 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:17:08.253 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:17:08.254254 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:17:18.315 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:17:18.313089 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:17:18.325 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:17:18.325496 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:17:28.398 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:17:28.398294 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:17:28.411 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:17:28.411707 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:17:38.479 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:17:38.480176 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:17:38.522 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:17:38.522819 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:17:48.613 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:17:48.613663 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:17:48.627 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:17:48.627842 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:17:58.749 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:17:58.749287 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:17:58.761 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:17:58.761876 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:18:08.825 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:18:08.825475 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:18:08.881 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:18:08.881773 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:18:19.028 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:18:19.029005 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:18:19.038 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:18:19.039236 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:18:29.141 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:18:29.142289 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:18:29.153 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:18:29.154279 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:18:39.257 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:18:39.257214 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:18:39.270 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:18:39.270593 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:18:49.372 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:18:49.372853 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:18:49.388 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:18:49.388905 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:18:59.485 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:18:59.485174 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:18:59.541 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:18:59.541873 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:19:09.649 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:19:09.645565 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:19:09.663 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:19:09.663332 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:19:19.781 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:19:19.781441 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:19:19.795 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:19:19.795922 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:19:29.903 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:19:29.903705 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:19:29.917 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:19:29.917878 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:19:40.018 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:19:40.018590 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:19:40.030 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:19:40.031126 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:19:50.108 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:19:50.108418 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:19:50.116 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:19:50.117563 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:20:00.229 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:20:00.229913 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:20:00.244 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:20:00.244665 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:20:10.357 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:20:10.357480 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:20:10.370 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:20:10.371040 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:20:20.474 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:20:20.474447 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:20:20.485 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:20:20.485991 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:20:30.565 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:20:30.566143 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:20:30.577 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:20:30.578203 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:20:40.652 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:20:40.652381 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:20:40.720 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:20:40.720719 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:20:50.835 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:20:50.834985 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:20:50.849 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:20:50.849690 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:21:00.971 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:21:00.971485 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:21:00.983 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:21:00.984108 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:21:11.084 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:21:11.084480 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:21:11.097 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:21:11.097676 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:21:21.203 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:21:21.203893 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:21:21.216 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:21:21.217568 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:21:31.295 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:21:31.295362 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:21:31.308 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:21:31.308839 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:21:41.430 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:21:41.431008 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:21:41.442 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:21:41.443491 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:21:51.552 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:21:51.552953 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:21:51.568 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:21:51.568298 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:22:01.647 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:22:01.648335 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:22:01.655 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:22:01.656594 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:22:11.723 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:22:11.723141 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:22:11.734 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:22:11.734656 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:22:21.810 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:22:21.810093 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:22:21.822 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:22:21.822704 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:22:31.899 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:22:31.900105 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:22:31.913 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:22:31.914047 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:22:42.018 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:22:42.018581 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:22:42.031 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:22:42.031591 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:22:52.134 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:22:52.135147 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:22:52.147 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:22:52.148269 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:23:02.224 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:23:02.224956 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:23:02.236 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:23:02.237500 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:23:12.316 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:23:12.316671 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:23:12.378 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:23:12.378924 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:23:22.460 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:23:22.460403 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:23:22.511 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:23:22.512134 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:23:32.585 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:23:32.586181 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:23:32.598 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:23:32.598987 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:23:42.665 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:23:42.665763 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:23:42.676 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:23:42.676988 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:23:52.747 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:23:52.747651 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:23:52.762 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:23:52.762899 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:24:02.887 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:24:02.888083 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:24:02.902 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:24:02.903389 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:24:12.978 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:24:12.979096 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:24:12.990 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:24:12.991388 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:24:23.109 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:24:23.109256 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:24:23.123 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:24:23.123923 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:24:33.266 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:24:33.266363 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:24:33.416 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:24:33.276689 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:24:43.400 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:24:43.400875 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:24:43.417 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:24:43.417129 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:24:53.539 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:24:53.540417 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:24:53.557 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:24:53.554747 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:25:03.667 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:25:03.667704 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:25:03.680 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:25:03.680877 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:25:13.817 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:25:13.817478 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:25:13.829 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:25:13.829839 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:25:23.905 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:25:23.906365 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:25:23.916 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:25:23.917090 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:25:34.094 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:25:34.094467 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:25:34.103 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:25:34.104541 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:25:44.205 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:25:44.205620 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:25:44.220 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:25:44.220700 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:25:54.343 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:25:54.343774 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:25:54.360 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:25:54.360833 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:26:04.456 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:26:04.457082 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:26:04.468 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:26:04.469446 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:26:14.535 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:26:14.534665 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:26:14.546 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:26:14.546641 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:26:24.652 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:26:24.652808 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:26:24.664 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:26:24.665149 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:26:34.732 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:26:34.733130 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:26:34.747 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:26:34.747788 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:26:44.816 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:26:44.816752 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:26:44.830 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:26:44.830913 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:26:54.931 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:26:54.931794 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:26:54.945 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:26:54.946263 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:27:05.058 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:27:05.059007 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:27:05.068 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:27:05.069341 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:27:15.131 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:27:15.132255 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:27:15.144 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:27:15.145117 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:27:25.224 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:27:25.224702 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:27:25.234 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:27:25.234768 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:27:35.303 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:27:35.303760 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:27:35.352 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:27:35.353160 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:27:45.448 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:27:45.448361 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:27:45.463 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:27:45.464247 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:27:55.553 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:27:55.553548 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:27:55.566 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:27:55.566863 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:28:05.750 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:28:05.751038 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:28:05.758 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:28:05.759165 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:28:15.984 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:28:15.985025 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:28:15.998 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:28:15.999373 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:28:26.173 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:28:26.173724 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:28:26.181 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:28:26.182311 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:28:36.392 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:28:36.392379 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:28:36.408 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:28:36.409364 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:28:46.469 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:28:46.469854 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:28:46.487 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:28:46.488076 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:28:56.571 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:28:56.571619 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:28:56.583 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:28:56.584182 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:29:06.659 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:29:06.659893 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:29:06.672 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:29:06.672852 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:29:16.787 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:29:16.787205 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:29:16.804 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:29:16.804485 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:29:26.879 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:29:26.879884 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:29:26.953 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:29:26.954345 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:29:37.064 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:29:37.065265 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:29:37.078 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:29:37.078834 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:29:47.195 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:29:47.195334 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:29:47.207 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:29:47.208230 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:29:57.311 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:29:57.311213 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:29:57.325 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:29:57.326142 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:30:00.137 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:30:00.138299 I | op-mon: parsing mon endpoints: a=10.110.141.123:6789,b=10.98.41.230:6789,c=10.109.47.142:6789 2022-03-09T01:30:00.784 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:30:00.784682 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-09T01:30:00.774+0000 7f417811d380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-09T01:30:00.909 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09T01:30:00.774+0000 7f417811d380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-09T01:30:00.909 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09T01:30:00.774+0000 7f417811d380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-09T01:30:00.910 INFO:tasks.rook.operator.smithi077.stdout:couldn't init storage provider. : exit status 5 2022-03-09T01:30:07.427 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:30:07.427401 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:30:07.440 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:30:07.441569 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:30:17.516 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:30:17.516385 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:30:17.528 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:30:17.529046 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:30:27.646 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:30:27.646813 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:30:27.658 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:30:27.659237 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:30:37.778 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:30:37.779044 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:30:37.790 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:30:37.791281 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:30:47.913 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:30:47.913711 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:30:47.924 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:30:47.924985 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:30:58.029 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:30:58.030388 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:30:58.041 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:30:58.041836 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:31:08.152 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:31:08.153036 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:31:08.161 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:31:08.161806 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:31:18.266 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:31:18.266528 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:31:18.279 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:31:18.279813 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:31:28.352 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:31:28.352275 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:31:28.364 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:31:28.365435 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:31:38.480 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:31:38.481139 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:31:38.496 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:31:38.496890 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:31:48.568 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:31:48.568837 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:31:48.770 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:31:48.633974 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:31:58.700 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:31:58.700655 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:31:58.713 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:31:58.714354 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:32:08.833 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:32:08.833726 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:32:08.845 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:32:08.845970 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:32:18.921 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:32:18.921753 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:32:18.980 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:32:18.981007 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:32:29.097 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:32:29.097680 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:32:29.111 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:32:29.111709 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:32:39.187 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:32:39.188026 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:32:39.263 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:32:39.263475 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:32:49.369 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:32:49.369640 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:32:49.383 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:32:49.383670 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:32:59.512 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:32:59.512718 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:32:59.525 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:32:59.525902 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:33:09.617 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:33:09.617882 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:33:09.631 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:33:09.632331 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:33:19.747 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:33:19.747272 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:33:19.764 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:33:19.761326 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:33:29.880 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:33:29.880449 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:33:29.892 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:33:29.892916 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:33:39.958 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:33:39.959005 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:33:39.971 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:33:39.972056 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:33:50.094 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:33:50.094717 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:33:50.126 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:33:50.126693 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:34:00.245 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:34:00.246175 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:34:00.259 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:34:00.260207 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:34:10.398 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:34:10.398627 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:34:10.415 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:34:10.415667 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:34:20.508 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:34:20.508791 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:34:20.520 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:34:20.521485 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:34:30.600 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:34:30.600883 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:34:30.614 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:34:30.614577 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:34:40.676 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:34:40.676666 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:34:40.687 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:34:40.688205 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:34:50.761 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:34:50.761727 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:34:50.776 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:34:50.776768 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:35:00.867 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:35:00.867706 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:35:00.881 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:35:00.881405 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:35:10.947 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:35:10.948152 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:35:10.958 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:35:10.959632 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:35:21.127 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:35:21.127468 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:35:21.137 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:35:21.138416 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:35:31.202 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:35:31.202415 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:35:31.213 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:35:31.213739 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:35:41.389 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:35:41.389421 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:35:41.400 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:35:41.401282 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:35:51.469 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:35:51.466907 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:35:51.476 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:35:51.476864 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:36:01.587 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:36:01.587146 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:36:01.597 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:36:01.598359 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:36:11.663 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:36:11.663504 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:36:11.701 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:36:11.702226 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:36:21.817 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:36:21.817513 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:36:21.828 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:36:21.828958 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:36:31.888 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:36:31.889188 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:36:31.901 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:36:31.901909 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:36:41.965 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:36:41.965778 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:36:41.977 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:36:41.977221 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:36:52.085 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:36:52.085800 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:36:52.100 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:36:52.100515 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:37:02.174 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:37:02.175032 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:37:02.188 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:37:02.188699 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:37:12.311 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:37:12.312135 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:37:12.323 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:37:12.324583 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:37:22.403 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:37:22.404322 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:37:22.417 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:37:22.417628 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:37:32.487 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:37:32.487622 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:37:32.502 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:37:32.502825 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:37:42.598 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:37:42.598574 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:37:42.609 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:37:42.610333 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:37:52.682 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:37:52.682974 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:37:52.692 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:37:52.692607 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:38:02.849 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:38:02.850142 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:38:02.861 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:38:02.862484 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:38:12.969 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:38:12.967732 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:38:12.979 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:38:12.979690 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:38:23.085 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:38:23.085184 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:38:23.096 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:38:23.096743 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:38:33.200 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:38:33.201201 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:38:33.215 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:38:33.215895 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:38:43.326 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:38:43.327338 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:38:43.340 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:38:43.341248 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:38:53.495 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:38:53.495664 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:38:53.505 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:38:53.505863 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:39:03.616 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:39:03.616958 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:39:03.630 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:39:03.631228 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:39:13.702 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:39:13.703457 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:39:13.712 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:39:13.712885 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:39:23.846 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:39:23.846801 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:39:23.862 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:39:23.862647 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:39:33.937 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:39:33.938306 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:39:33.949 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:39:33.950105 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:39:44.035 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:39:44.036511 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:39:44.052 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:39:44.052304 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:39:54.173 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:39:54.173731 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:39:54.183 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:39:54.184206 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:40:04.256 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:40:04.256659 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:40:04.304 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:40:04.305248 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:40:14.413 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:40:14.413627 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:40:14.423 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:40:14.424429 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:40:24.523 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:40:24.523647 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:40:24.539 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:40:24.536644 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:40:34.683 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:40:34.684026 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:40:34.693 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:40:34.694196 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:40:44.832 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:40:44.832518 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:40:44.843 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:40:44.843829 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:40:54.941 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:40:54.941810 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:40:54.953 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:40:54.954057 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:41:05.070 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:41:05.071028 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:41:05.083 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:41:05.083754 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:41:15.149 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:41:15.149616 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:41:15.160 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:41:15.161181 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:41:25.235 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:41:25.235935 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:41:25.248 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:41:25.249172 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:41:35.318 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:41:35.318839 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:41:35.385 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:41:35.385501 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:41:45.490 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:41:45.491147 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:41:45.501 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:41:45.502453 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:41:55.560 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:41:55.561008 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:41:55.572 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:41:55.573175 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:42:05.683 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:42:05.684077 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:42:05.698 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:42:05.698667 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:42:15.804 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:42:15.805100 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:42:15.816 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:42:15.817046 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:42:25.884 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:42:25.884578 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:42:25.897 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:42:25.898177 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:42:35.971 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:42:35.971533 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:42:35.983 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:42:35.984102 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:42:46.090 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:42:46.091161 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:42:46.103 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:42:46.103674 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:42:56.168 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:42:56.168615 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:42:56.249 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:42:56.249514 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:43:06.321 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:43:06.322346 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:43:06.335 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:43:06.335706 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:43:16.443 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:43:16.443649 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:43:16.452 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:43:16.453080 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:43:26.533 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:43:26.533395 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:43:26.544 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:43:26.545254 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:43:36.612 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:43:36.613055 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:43:36.655 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:43:36.654555 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:43:46.724 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:43:46.725205 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:43:46.738 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:43:46.738824 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:43:56.857 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:43:56.857021 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:43:56.865 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:43:56.865894 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:44:06.935 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:44:06.936220 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:44:06.986 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:44:06.986342 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:44:17.124 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:44:17.125209 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:44:17.139 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:44:17.139610 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:44:27.240 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:44:27.240682 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:44:27.253 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:44:27.254046 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:44:37.370 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:44:37.370894 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:44:37.383 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:44:37.383839 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:44:47.470 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:44:47.470587 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:44:47.486 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:44:47.486657 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:44:57.585 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:44:57.585615 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:44:57.598 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:44:57.598887 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:45:07.673 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:45:07.673840 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:45:07.696 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:45:07.697106 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:45:17.782 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:45:17.782453 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:45:17.794 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:45:17.794922 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:45:27.879 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:45:27.879745 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:45:27.891 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:45:27.891891 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:45:38.011 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:45:38.012208 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:45:38.026 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:45:38.026864 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:45:48.168 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:45:48.169021 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:45:48.181 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:45:48.182228 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:45:58.250 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:45:58.250952 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:45:58.263 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:45:58.263924 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:46:08.330 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:46:08.328422 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:46:08.341 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:46:08.342169 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:46:18.433 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:46:18.433598 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:46:18.443 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:46:18.444397 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:46:28.586 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:46:28.587112 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:46:28.601 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:46:28.601612 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:46:38.698 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:46:38.699162 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:46:38.711 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:46:38.712704 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:46:40.799 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:46:40.799641 I | op-mon: parsing mon endpoints: a=10.110.141.123:6789,b=10.98.41.230:6789,c=10.109.47.142:6789 2022-03-09T01:46:41.463 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:46:41.463364 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-09T01:46:41.453+0000 7f9f6475a380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-09T01:46:41.463 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09T01:46:41.453+0000 7f9f6475a380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-09T01:46:41.464 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09T01:46:41.453+0000 7f9f6475a380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-09T01:46:41.464 INFO:tasks.rook.operator.smithi077.stdout:couldn't init storage provider. : exit status 5 2022-03-09T01:46:48.837 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:46:48.838018 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:46:48.849 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:46:48.850475 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:46:58.967 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:46:58.967728 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:46:58.982 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:46:58.983107 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:47:09.058 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:47:09.058269 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:47:09.072 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:47:09.072900 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:47:19.168 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:47:19.169057 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:47:19.179 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:47:19.180407 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:47:29.283 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:47:29.283843 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:47:29.296 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:47:29.297526 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:47:39.435 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:47:39.435948 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:47:39.449 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:47:39.450127 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:47:49.535 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:47:49.535700 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:47:49.548 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:47:49.549586 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:47:59.656 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:47:59.657191 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:47:59.668 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:47:59.668784 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:48:09.733 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:48:09.734043 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:48:09.741 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:48:09.742434 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:48:19.849 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:48:19.850051 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:48:19.861 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:48:19.861928 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:48:29.939 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:48:29.940337 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:48:29.953 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:48:29.954472 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:48:40.025 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:48:40.025454 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:48:40.052 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:48:40.051416 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:48:50.158 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:48:50.159222 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:48:50.171 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:48:50.172368 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:49:00.362 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:49:00.363110 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:49:00.373 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:49:00.373903 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:49:10.450 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:49:10.450599 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:49:10.586 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:49:10.586876 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:49:20.650 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:49:20.651153 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:49:20.664 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:49:20.665516 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:49:30.784 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:49:30.785377 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:49:30.797 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:49:30.798137 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:49:40.892 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:49:40.892323 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:49:40.906 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:49:40.906953 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:49:50.973 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:49:50.973704 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:49:50.984 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:49:50.985543 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:50:01.056 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:50:01.057270 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:50:01.071 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:50:01.072008 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:50:11.181 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:50:11.181826 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:50:11.194 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:50:11.195382 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:50:21.273 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:50:21.273793 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:50:21.285 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:50:21.286044 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:50:31.348 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:50:31.348426 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:50:31.361 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:50:31.361852 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:50:41.485 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:50:41.486429 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:50:41.501 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:50:41.502331 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:50:51.584 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:50:51.584459 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:50:51.596 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:50:51.596596 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:51:01.664 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:51:01.665108 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:51:01.677 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:51:01.677894 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:51:11.762 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:51:11.762595 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:51:11.817 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:51:11.818031 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:51:21.923 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:51:21.924088 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:51:21.935 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:51:21.936375 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:51:32.045 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:51:32.045277 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:51:32.057 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:51:32.058214 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:51:42.139 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:51:42.139602 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:51:42.150 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:51:42.151038 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:51:52.213 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:51:52.214313 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:51:52.225 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:51:52.225907 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:52:02.290 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:52:02.291041 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:52:02.303 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:52:02.304325 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:52:12.371 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:52:12.371776 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:52:12.383 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:52:12.384459 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:52:22.535 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:52:22.535423 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:52:22.548 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:52:22.548887 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:52:32.646 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:52:32.646906 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:52:32.660 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:52:32.660962 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:52:42.770 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:52:42.770726 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:52:42.783 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:52:42.784567 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:52:52.896 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:52:52.897083 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:52:52.911 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:52:52.912566 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:53:03.020 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:53:03.020741 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:53:03.032 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:53:03.032882 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:53:13.160 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:53:13.160987 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:53:13.172 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:53:13.173594 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:53:23.279 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:53:23.280078 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:53:23.292 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:53:23.293371 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:53:33.371 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:53:33.371413 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:53:33.379 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:53:33.380330 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:53:43.477 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:53:43.477934 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:53:43.562 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:53:43.563082 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:53:53.675 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:53:53.675654 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:53:53.685 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:53:53.686444 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:54:03.808 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:54:03.808683 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:54:03.822 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:54:03.822384 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:54:13.966 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:54:13.966192 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:54:13.979 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:54:13.980240 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:54:24.056 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:54:24.057386 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:54:24.069 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:54:24.069903 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:54:34.172 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:54:34.172576 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:54:34.184 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:54:34.184598 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:54:44.287 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:54:44.288313 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:54:44.301 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:54:44.302244 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:54:54.422 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:54:54.422469 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:54:54.435 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:54:54.435657 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:55:04.503 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:55:04.503545 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:55:04.516 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:55:04.516772 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:55:14.658 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:55:14.658690 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:55:14.670 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:55:14.671199 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:55:24.748 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:55:24.749164 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:55:24.766 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:55:24.766610 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:55:34.901 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:55:34.902300 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:55:34.915 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:55:34.916627 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:55:45.016 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:55:45.016698 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:55:45.027 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:55:45.028477 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:55:55.157 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:55:55.157807 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:55:55.178 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:55:55.178934 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:56:05.324 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:56:05.325320 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:56:05.336 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:56:05.337252 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:56:15.414 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:56:15.414339 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:56:15.425 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:56:15.426662 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:56:25.522 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:56:25.522571 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:56:25.538 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:56:25.535611 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:56:35.605 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:56:35.606401 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:56:35.630 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:56:35.631252 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:56:45.735 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:56:45.736424 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:56:45.747 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:56:45.747800 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:56:55.877 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:56:55.877862 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:56:55.888 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:56:55.889421 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:57:06.001 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:57:06.001949 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:57:06.016 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:57:06.017524 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:57:16.134 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:57:16.134527 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:57:16.148 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:57:16.149242 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:57:26.215 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:57:26.215623 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:57:26.230 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:57:26.231254 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:57:36.299 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:57:36.299981 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:57:36.367 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:57:36.368338 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:57:46.455 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:57:46.455556 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:57:46.526 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:57:46.527499 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:57:56.599 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:57:56.600381 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:57:56.613 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:57:56.614073 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:58:06.678 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:58:06.678888 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:58:06.753 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:58:06.754218 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:58:16.827 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:58:16.827649 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:58:16.840 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:58:16.840969 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:58:26.940 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:58:26.940801 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:58:26.953 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:58:26.954828 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:58:37.072 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:58:37.072910 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:58:37.085 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:58:37.086580 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:58:47.203 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:58:47.204264 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:58:47.217 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:58:47.218643 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:58:57.370 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:58:57.365938 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:58:57.379 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:58:57.379128 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:59:07.523 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:59:07.523751 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:59:07.537 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:59:07.538593 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:59:17.617 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:59:17.618585 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:59:17.660 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:59:17.661196 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:59:27.740 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:59:27.741412 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:59:27.753 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:59:27.754632 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:59:37.882 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:59:37.882778 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:59:37.895 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:59:37.896297 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:59:48.013 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:59:48.013913 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:59:48.024 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:59:48.024942 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:59:58.142 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:59:58.143292 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:59:58.156 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 01:59:58.157861 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:00:08.268 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:00:08.263768 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:00:08.283 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:00:08.284739 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:00:18.358 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:00:18.359487 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:00:18.371 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:00:18.372211 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:00:28.444 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:00:28.445104 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:00:28.459 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:00:28.459853 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:00:38.520 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:00:38.520897 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:00:38.529 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:00:38.530249 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:00:48.659 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:00:48.660023 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:00:48.672 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:00:48.672900 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:00:58.766 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:00:58.766870 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:00:58.779 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:00:58.780403 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:01:08.851 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:01:08.852208 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:01:08.922 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:01:08.923250 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:01:19.036 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:01:19.036775 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:01:19.048 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:01:19.049588 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:01:29.128 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:01:29.114974 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:01:29.193 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:01:29.194428 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:01:39.266 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:01:39.267133 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:01:39.319 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:01:39.319648 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:01:49.443 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:01:49.443648 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:01:49.456 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:01:49.456924 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:01:59.575 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:01:59.576139 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:01:59.588 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:01:59.589785 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:02:09.706 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:02:09.707448 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:02:09.722 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:02:09.722843 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:02:19.808 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:02:19.808710 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:02:19.821 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:02:19.822050 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:02:29.893 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:02:29.894105 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:02:29.980 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:02:29.980543 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:02:40.221 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:02:40.222011 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:02:40.250 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:02:40.251421 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:02:50.391 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:02:50.391850 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:02:50.404 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:02:50.405083 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:03:00.490 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:03:00.490702 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:03:00.558 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:03:00.558851 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:03:10.630 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:03:10.631252 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:03:10.642 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:03:10.643468 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:03:20.735 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:03:20.736311 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:03:20.748 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:03:20.749102 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:03:21.470 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:03:21.471292 I | op-mon: parsing mon endpoints: a=10.110.141.123:6789,b=10.98.41.230:6789,c=10.109.47.142:6789 2022-03-09T02:03:22.131 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:03:22.132016 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-09T02:03:22.120+0000 7f9c1ee39380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-09T02:03:22.132 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09T02:03:22.120+0000 7f9c1ee39380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-09T02:03:22.132 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09T02:03:22.120+0000 7f9c1ee39380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-09T02:03:22.132 INFO:tasks.rook.operator.smithi077.stdout:couldn't init storage provider. : exit status 5 2022-03-09T02:03:30.821 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:03:30.822106 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:03:30.832 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:03:30.833748 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:03:40.937 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:03:40.938122 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:03:40.949 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:03:40.950234 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:03:51.049 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:03:51.050162 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:03:51.061 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:03:51.062763 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:04:01.167 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:04:01.167755 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:04:01.182 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:04:01.183045 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:04:11.308 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:04:11.308561 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:04:11.320 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:04:11.321130 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:04:21.382 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:04:21.382825 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:04:21.441 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:04:21.442503 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:04:31.540 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:04:31.540592 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:04:31.555 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:04:31.556006 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:04:41.646 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:04:41.647424 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:04:41.658 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:04:41.659436 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:04:51.726 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:04:51.727517 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:04:51.739 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:04:51.740324 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:05:01.803 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:05:01.803440 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:05:01.815 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:05:01.816728 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:05:11.933 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:05:11.934231 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:05:11.943 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:05:11.944665 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:05:22.011 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:05:22.012510 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:05:22.076 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:05:22.077550 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:05:32.171 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:05:32.172109 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:05:32.185 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:05:32.186149 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:05:42.289 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:05:42.289893 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:05:42.304 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:05:42.305208 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:05:52.495 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:05:52.492419 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:05:52.501 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:05:52.502623 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:06:02.656 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:06:02.657503 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:06:02.670 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:06:02.671774 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:06:12.739 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:06:12.740273 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:06:12.813 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:06:12.814053 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:06:22.957 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:06:22.958152 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:06:22.971 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:06:22.972598 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:06:33.042 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:06:33.042692 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:06:33.055 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:06:33.056126 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:06:43.170 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:06:43.171022 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:06:43.283 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:06:43.186288 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:06:53.255 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:06:53.256131 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:06:53.268 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:06:53.269746 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:07:03.391 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:07:03.392251 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:07:03.404 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:07:03.405335 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:07:13.515 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:07:13.516103 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:07:13.526 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:07:13.527209 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:07:23.659 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:07:23.660123 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:07:23.671 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:07:23.672877 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:07:33.767 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:07:33.767855 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:07:33.779 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:07:33.779886 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:07:43.920 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:07:43.920727 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:07:43.933 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:07:43.934136 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:07:54.030 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:07:54.031302 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:07:54.043 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:07:54.044523 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:08:04.149 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:08:04.150226 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:08:04.166 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:08:04.166529 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:08:14.265 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:08:14.266026 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:08:14.278 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:08:14.279060 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:08:24.375 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:08:24.376452 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:08:24.435 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:08:24.436511 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:08:34.575 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:08:34.575752 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:08:34.590 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:08:34.587143 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:08:44.732 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:08:44.733613 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:08:44.743 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:08:44.744631 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:08:54.894 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:08:54.894684 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:08:54.905 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:08:54.906482 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:09:05.187 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:09:05.187733 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:09:05.199 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:09:05.200309 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:09:15.297 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:09:15.297477 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:09:15.305 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:09:15.306280 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:09:25.389 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:09:25.390370 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:09:25.400 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:09:25.401124 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:09:35.513 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:09:35.513731 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:09:35.522 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:09:35.523548 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:09:45.589 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:09:45.586990 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:09:45.599 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:09:45.600865 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:09:55.660 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:09:55.657150 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:09:55.669 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:09:55.670678 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:10:05.735 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:10:05.736221 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:10:05.745 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:10:05.746438 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:10:15.810 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:10:15.811533 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:10:15.822 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:10:15.823590 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:10:25.946 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:10:25.947071 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:10:25.957 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:10:25.958201 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:10:36.059 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:10:36.060109 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:10:36.072 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:10:36.073394 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:10:46.234 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:10:46.234926 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:10:46.247 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:10:46.248395 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:10:56.339 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:10:56.339506 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:10:56.351 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:10:56.352255 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:11:06.419 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:11:06.420445 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:11:06.490 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:11:06.491117 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:11:16.556 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:11:16.556718 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:11:16.567 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:11:16.568766 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:11:26.637 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:11:26.637411 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:11:26.645 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:11:26.646842 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:11:36.719 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:11:36.719573 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:11:36.730 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:11:36.731544 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:11:46.902 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:11:46.903283 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:11:46.912 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:11:46.913675 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:11:57.020 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:11:57.021554 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:11:57.032 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:11:57.033582 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:12:07.158 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:12:07.158939 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:12:07.171 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:12:07.172163 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:12:17.276 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:12:17.276947 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:12:17.288 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:12:17.288532 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:12:27.372 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:12:27.372908 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:12:27.385 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:12:27.386370 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:12:37.529 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:12:37.530096 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:12:37.539 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:12:37.540653 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:12:47.635 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:12:47.636311 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:12:47.648 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:12:47.649433 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:12:57.816 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:12:57.817106 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:12:57.844 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:12:57.845328 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:13:07.995 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:13:07.996399 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:13:08.008 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:13:08.009407 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:13:18.145 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:13:18.145986 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:13:18.156 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:13:18.156496 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:13:28.280 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:13:28.281351 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:13:28.290 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:13:28.291448 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:13:38.356 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:13:38.356865 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:13:38.366 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:13:38.367038 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:13:48.466 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:13:48.466891 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:13:48.476 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:13:48.477789 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:13:58.573 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:13:58.573798 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:13:58.586 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:13:58.587746 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:14:08.689 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:14:08.689644 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:14:08.701 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:14:08.702094 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:14:18.825 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:14:18.825977 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:14:18.838 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:14:18.839433 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:14:28.911 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:14:28.911872 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:14:28.925 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:14:28.926702 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:14:39.045 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:14:39.046565 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:14:39.060 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:14:39.061586 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:14:49.126 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:14:49.127154 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:14:49.137 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:14:49.138462 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:14:59.244 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:14:59.244619 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:14:59.256 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:14:59.257227 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:15:09.328 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:15:09.329126 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:15:09.340 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:15:09.341113 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:15:19.444 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:15:19.445373 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:15:19.459 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:15:19.460547 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:15:29.590 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:15:29.591082 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:15:29.604 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:15:29.605015 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:15:39.697 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:15:39.697621 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:15:39.759 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:15:39.760537 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:15:49.860 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:15:49.860690 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:15:49.874 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:15:49.875304 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:16:00.000 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:16:00.001303 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:16:00.027 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:16:00.028275 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:16:10.116 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:16:10.117530 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:16:10.129 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:16:10.130621 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:16:20.245 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:16:20.246131 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:16:20.254 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:16:20.255490 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:16:30.345 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:16:30.345765 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:16:30.358 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:16:30.359719 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:16:40.423 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:16:40.424191 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:16:40.434 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:16:40.435626 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:16:50.514 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:16:50.514995 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:16:50.526 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:16:50.527500 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:17:00.634 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:17:00.634649 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:17:00.684 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:17:00.684627 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:17:10.795 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:17:10.795855 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:17:10.812 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:17:10.809137 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:17:20.884 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:17:20.884874 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:17:20.897 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:17:20.898323 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:17:31.007 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:17:31.007878 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:17:31.020 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:17:31.021646 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:17:41.139 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:17:41.140104 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:17:41.152 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:17:41.153639 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:17:51.249 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:17:51.249921 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:17:51.260 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:17:51.261729 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:18:01.368 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:18:01.368888 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:18:01.378 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:18:01.379738 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:18:11.446 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:18:11.447623 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:18:11.459 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:18:11.460310 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:18:21.598 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:18:21.599508 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:18:21.609 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:18:21.610506 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:18:31.716 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:18:31.716905 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:18:31.730 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:18:31.731456 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:18:41.798 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:18:41.798537 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:18:41.811 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:18:41.812228 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:18:51.936 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:18:51.936802 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:18:51.950 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:18:51.951056 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:19:02.019 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:19:02.019507 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:19:02.031 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:19:02.032765 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:19:12.126 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:19:12.126885 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:19:12.142 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:19:12.143679 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:19:22.254 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:19:22.255159 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:19:22.266 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:19:22.266846 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:19:32.378 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:19:32.378823 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:19:32.391 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:19:32.392166 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:19:42.500 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:19:42.501140 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:19:42.512 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:19:42.513708 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:19:52.586 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:19:52.586693 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:19:52.597 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:19:52.598575 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:20:02.140 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:20:02.141056 I | op-mon: parsing mon endpoints: a=10.110.141.123:6789,b=10.98.41.230:6789,c=10.109.47.142:6789 2022-03-09T02:20:02.711 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:20:02.711833 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:20:02.727 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:20:02.728803 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:20:02.777 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:20:02.777695 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-09T02:20:02.771+0000 7f8827c36380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-09T02:20:02.777 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09T02:20:02.771+0000 7f8827c36380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-09T02:20:02.778 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09T02:20:02.771+0000 7f8827c36380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-09T02:20:02.778 INFO:tasks.rook.operator.smithi077.stdout:couldn't init storage provider. : exit status 5 2022-03-09T02:20:12.917 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:20:12.917614 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:20:12.927 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:20:12.928185 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:20:23.085 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:20:23.085760 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:20:23.095 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:20:23.096590 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:20:33.223 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:20:33.224531 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:20:33.236 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:20:33.237508 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:20:43.370 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:20:43.371497 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:20:43.382 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:20:43.383781 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:20:53.530 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:20:53.531397 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:20:53.544 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:20:53.545040 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:21:03.621 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:21:03.622568 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:21:03.635 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:21:03.636536 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:21:13.763 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:21:13.763783 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:21:13.776 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:21:13.776937 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:21:23.904 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:21:23.905350 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:21:23.916 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:21:23.917182 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:21:34.085 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:21:34.086002 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:21:34.581 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:21:34.098207 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:21:44.204 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:21:44.205331 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:21:44.216 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:21:44.217721 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:21:54.314 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:21:54.314604 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:21:54.327 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:21:54.328802 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:22:04.427 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:22:04.428163 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:22:04.439 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:22:04.440372 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:22:14.548 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:22:14.548675 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:22:14.559 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:22:14.560098 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:22:24.641 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:22:24.641673 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:22:24.706 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:22:24.706968 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:22:34.787 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:22:34.788115 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:22:34.802 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:22:34.803055 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:22:44.875 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:22:44.876094 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:22:45.039 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:22:45.040368 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:22:55.170 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:22:55.170887 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:22:55.179 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:22:55.180536 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:23:05.279 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:23:05.279652 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:23:05.293 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:23:05.294759 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:23:15.364 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:23:15.365169 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:23:15.467 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:23:15.467658 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:23:25.604 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:23:25.605287 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:23:25.615 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:23:25.616016 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:23:35.693 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:23:35.693704 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:23:35.705 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:23:35.706029 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:23:45.765 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:23:45.766281 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:23:45.773 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:23:45.774514 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:23:55.925 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:23:55.925687 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:23:55.941 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:23:55.941821 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:24:06.012 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:24:06.013111 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:24:06.024 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:24:06.025863 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:24:16.140 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:24:16.140740 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:24:16.152 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:24:16.153178 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:24:26.263 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:24:26.264357 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:24:26.276 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:24:26.276834 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:24:36.384 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:24:36.385192 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:24:36.398 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:24:36.398749 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:24:46.469 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:24:46.469550 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:24:46.478 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:24:46.479844 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:24:56.578 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:24:56.579400 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:24:56.594 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:24:56.595482 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:25:06.702 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:25:06.702652 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:25:06.715 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:25:06.716136 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:25:16.792 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:25:16.793083 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:25:16.806 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:25:16.807559 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:25:26.910 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:25:26.910642 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:25:26.919 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:25:26.920101 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:25:36.989 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:25:36.990443 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:25:36.999 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:25:37.000115 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:25:47.131 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:25:47.132375 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:25:47.144 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:25:47.145628 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:25:57.263 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:25:57.263440 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:25:57.276 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:25:57.277452 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:26:07.384 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:26:07.385203 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:26:07.399 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:26:07.400494 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:26:17.507 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:26:17.507649 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:26:17.520 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:26:17.521451 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:26:27.600 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:26:27.600937 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:26:27.615 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:26:27.616238 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:26:37.680 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:26:37.678373 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:26:37.690 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:26:37.691529 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:26:47.764 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:26:47.764368 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:26:47.777 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:26:47.777854 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:26:57.857 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:26:57.857633 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:26:57.865 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:26:57.866451 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:27:07.972 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:27:07.972798 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:27:07.986 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:27:07.987291 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:27:18.101 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:27:18.101738 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:27:18.113 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:27:18.114608 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:27:28.232 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:27:28.232925 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:27:28.240 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:27:28.241783 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:27:38.388 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:27:38.389304 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:27:38.401 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:27:38.402543 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:27:48.513 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:27:48.513640 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:27:48.525 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:27:48.526644 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:27:58.595 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:27:58.595727 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:27:58.609 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:27:58.610650 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:28:08.682 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:28:08.682759 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:28:08.743 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:28:08.693834 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:28:18.809 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:28:18.809845 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:28:18.822 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:28:18.823123 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:28:28.915 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:28:28.915618 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:28:28.932 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:28:28.929090 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:28:39.278 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:28:39.278621 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:28:39.372 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:28:39.290879 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:28:49.358 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:28:49.358342 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:28:49.369 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:28:49.369888 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:28:59.508 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:28:59.508705 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:28:59.521 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:28:59.522007 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:29:09.583 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:29:09.581313 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:29:09.633 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:29:09.633946 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:29:19.743 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:29:19.744364 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:29:19.757 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:29:19.758449 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:29:29.866 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:29:29.867445 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:29:29.874 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:29:29.875718 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:29:39.999 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:29:39.999842 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:29:40.020 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:29:40.020834 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:29:50.104 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:29:50.104752 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:29:50.117 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:29:50.117801 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:30:00.180 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:30:00.180772 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:30:00.190 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:30:00.191075 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:30:10.315 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:30:10.315638 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:30:10.327 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:30:10.328028 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:30:20.398 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:30:20.399000 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:30:20.408 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:30:20.409294 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:30:30.517 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:30:30.518201 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:30:30.535 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:30:30.532349 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:30:40.673 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:30:40.674209 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:30:40.686 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:30:40.686666 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:30:50.782 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:30:50.782502 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:30:50.796 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:30:50.797146 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:31:00.909 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:31:00.909904 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:31:00.920 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:31:00.921622 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:31:10.981 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:31:10.982320 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:31:10.996 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:31:10.997371 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:31:21.086 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:31:21.086593 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:31:21.100 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:31:21.101090 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:31:31.205 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:31:31.205880 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:31:31.217 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:31:31.218258 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:31:41.318 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:31:41.318656 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:31:41.331 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:31:41.331910 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:31:51.428 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:31:51.428402 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:31:51.441 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:31:51.442198 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:32:01.548 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:32:01.548981 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:32:01.590 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:32:01.590997 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:32:11.658 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:32:11.658752 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:32:11.670 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:32:11.671562 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:32:21.738 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:32:21.738296 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:32:21.752 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:32:21.752528 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:32:31.853 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:32:31.853932 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:32:31.866 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:32:31.866797 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:32:41.950 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:32:41.950713 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:32:41.962 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:32:41.963472 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:32:52.031 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:32:52.031528 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:32:52.043 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:32:52.043992 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:33:02.148 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:33:02.149072 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:33:02.161 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:33:02.162005 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:33:12.232 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:33:12.233322 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:33:12.248 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:33:12.249197 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:33:22.362 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:33:22.362198 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:33:22.374 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:33:22.375284 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:33:32.450 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:33:32.450532 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:33:32.465 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:33:32.466069 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:33:42.529 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:33:42.529792 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:33:42.542 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:33:42.543273 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:33:52.643 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:33:52.643592 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:33:52.657 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:33:52.657611 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:34:02.727 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:34:02.727620 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:34:02.742 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:34:02.742969 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:34:12.804 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:34:12.804581 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:34:12.816 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:34:12.817444 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:34:22.935 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:34:22.935599 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:34:22.948 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:34:22.948713 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:34:33.052 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:34:33.053016 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:34:33.066 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:34:33.066877 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:34:43.171 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:34:43.172064 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:34:43.182 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:34:43.182823 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:34:53.299 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:34:53.299632 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:34:53.310 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:34:53.311113 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:35:03.417 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:35:03.416955 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:35:03.446 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:35:03.446892 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:35:13.568 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:35:13.568724 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:35:13.582 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:35:13.583342 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:35:23.758 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:35:23.758281 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:35:23.769 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:35:23.769764 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:35:33.897 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:35:33.897613 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:35:33.907 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:35:33.907873 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:35:43.980 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:35:43.980499 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:35:44.025 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:35:44.025957 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:35:54.129 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:35:54.129404 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:35:54.145 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:35:54.145801 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:36:04.232 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:36:04.232508 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:36:04.244 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:36:04.245234 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:36:14.340 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:36:14.340423 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:36:14.352 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:36:14.352520 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:36:24.466 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:36:24.466309 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:36:24.475 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:36:24.475713 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:36:34.584 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:36:34.584408 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:36:34.595 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:36:34.595760 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:36:42.786 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:36:42.786883 I | op-mon: parsing mon endpoints: a=10.110.141.123:6789,b=10.98.41.230:6789,c=10.109.47.142:6789 2022-03-09T02:36:43.507 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:36:43.507494 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-09T02:36:43.494+0000 7f6d324e4380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-09T02:36:43.508 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09T02:36:43.494+0000 7f6d324e4380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-09T02:36:43.508 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09T02:36:43.494+0000 7f6d324e4380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-09T02:36:43.508 INFO:tasks.rook.operator.smithi077.stdout:couldn't init storage provider. : exit status 5 2022-03-09T02:36:44.696 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:36:44.696772 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:36:44.708 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:36:44.708902 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:36:54.827 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:36:54.827293 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:36:54.841 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:36:54.842375 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:37:04.935 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:37:04.935531 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:37:04.946 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:37:04.946907 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:37:15.087 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:37:15.087906 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:37:15.100 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:37:15.100946 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:37:25.187 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:37:25.187709 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:37:25.266 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:37:25.266457 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:37:35.339 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:37:35.339157 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:37:35.352 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:37:35.352463 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:37:45.457 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:37:45.457934 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:37:45.467 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:37:45.468365 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:37:55.564 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:37:55.565178 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:37:55.577 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:37:55.577871 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:38:05.638 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:38:05.638567 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:38:05.648 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:38:05.649039 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:38:15.710 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:38:15.710717 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:38:15.719 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:38:15.720464 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:38:25.793 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:38:25.794042 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:38:25.866 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:38:25.865444 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:38:35.939 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:38:35.939544 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:38:35.948 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:38:35.948522 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:38:46.010 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:38:46.010786 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:38:46.022 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:38:46.022822 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:38:56.143 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:38:56.143626 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:38:56.156 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:38:56.157071 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:39:06.261 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:39:06.261125 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:39:06.271 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:39:06.272227 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:39:16.350 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:39:16.350823 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:39:16.368 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:39:16.368738 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:39:26.479 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:39:26.479056 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:39:26.488 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:39:26.488881 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:39:36.550 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:39:36.550935 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:39:36.562 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:39:36.563221 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:39:46.666 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:39:46.666349 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:39:46.678 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:39:46.678095 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:39:56.743 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:39:56.743772 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:39:56.755 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:39:56.755828 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:40:06.883 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:40:06.883399 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:40:06.952 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:40:06.895524 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:40:17.000 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:40:17.000403 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:40:17.011 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:40:17.012012 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:40:27.121 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:40:27.121271 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:40:27.132 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:40:27.132480 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:40:37.266 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:40:37.266904 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:40:37.278 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:40:37.278636 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:40:47.501 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:40:47.501649 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:40:47.575 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:40:47.575119 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:40:57.655 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:40:57.655664 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:40:57.668 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:40:57.668948 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:41:07.772 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:41:07.772833 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:41:07.785 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:41:07.785790 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:41:17.854 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:41:17.854208 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:41:17.872 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:41:17.869097 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:41:27.993 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:41:27.993129 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:41:28.004 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:41:28.005363 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:41:38.134 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:41:38.134763 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:41:38.145 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:41:38.145543 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:41:48.217 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:41:48.217023 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:41:48.229 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:41:48.230242 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:41:58.321 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:41:58.321820 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:41:58.335 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:41:58.335230 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:42:08.430 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:42:08.430590 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:42:08.443 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:42:08.443817 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:42:18.554 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:42:18.553937 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:42:18.565 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:42:18.566211 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:42:28.678 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:42:28.678633 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:42:28.689 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:42:28.690174 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:42:38.757 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:42:38.757666 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:42:38.770 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:42:38.770671 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:42:48.842 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:42:48.842289 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:42:48.853 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:42:48.853711 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:42:58.916 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:42:58.916587 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:42:58.932 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:42:58.929271 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:43:09.039 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:43:09.039555 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:43:09.053 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:43:09.053274 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:43:19.128 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:43:19.128125 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:43:19.203 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:43:19.202936 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:43:29.307 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:43:29.307568 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:43:29.321 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:43:29.321438 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:43:39.395 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:43:39.395071 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:43:39.408 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:43:39.408207 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:43:49.516 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:43:49.516614 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:43:49.530 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:43:49.530447 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:43:59.612 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:43:59.612807 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:43:59.624 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:43:59.624173 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:44:09.735 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:44:09.735376 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:44:09.746 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:44:09.746880 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:44:19.822 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:44:19.822910 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:44:19.834 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:44:19.835012 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:44:29.953 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:44:29.953854 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:44:29.966 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:44:29.967086 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:44:40.044 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:44:40.044761 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:44:40.125 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:44:40.125983 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:44:50.254 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:44:50.253868 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:44:50.266 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:44:50.266785 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:45:00.381 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:45:00.381419 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:45:00.400 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:45:00.401273 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:45:10.527 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:45:10.527610 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:45:10.541 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:45:10.541970 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:45:20.668 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:45:20.668024 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:45:20.680 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:45:20.680884 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:45:30.815 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:45:30.814894 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:45:30.828 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:45:30.828089 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:45:40.955 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:45:40.954945 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:45:40.967 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:45:40.967863 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:45:51.074 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:45:51.074277 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:45:51.088 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:45:51.088770 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:46:01.223 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:46:01.223249 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:46:01.236 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:46:01.236956 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:46:11.306 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:46:11.306611 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:46:11.325 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:46:11.323754 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:46:21.435 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:46:21.435346 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:46:21.448 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:46:21.447448 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:46:31.578 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:46:31.578489 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:46:31.603 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:46:31.603173 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:46:41.693 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:46:41.693033 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:46:41.706 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:46:41.706320 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:46:51.772 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:46:51.771890 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:46:51.786 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:46:51.786231 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:47:01.918 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:47:01.918683 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:47:01.933 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:47:01.933928 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:47:11.992 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:47:11.989461 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:47:12.002 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:47:12.002345 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:47:22.097 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:47:22.097952 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:47:22.111 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:47:22.111074 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:47:32.214 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:47:32.214358 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:47:32.228 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:47:32.228264 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:47:42.304 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:47:42.304000 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:47:42.316 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:47:42.316534 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:47:52.422 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:47:52.422057 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:47:52.435 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:47:52.435213 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:48:02.548 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:48:02.548382 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:48:02.562 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:48:02.562111 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:48:12.663 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:48:12.663723 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:48:12.675 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:48:12.675394 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:48:22.818 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:48:22.818016 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:48:22.830 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:48:22.830738 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:48:32.891 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:48:32.891568 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:48:32.904 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:48:32.904301 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:48:43.014 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:48:43.014124 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:48:43.033 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:48:43.029243 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:48:53.119 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:48:53.118884 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:48:53.131 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:48:53.131153 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:49:03.198 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:49:03.198108 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:49:03.211 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:49:03.211048 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:49:13.298 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:49:13.298806 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:49:13.310 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:49:13.310032 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:49:23.425 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:49:23.424992 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:49:23.436 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:49:23.436955 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:49:33.574 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:49:33.574173 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:49:33.586 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:49:33.585763 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:49:43.651 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:49:43.651160 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:49:43.662 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:49:43.662230 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:49:53.723 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:49:53.723580 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:49:53.737 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:49:53.737864 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:50:03.888 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:50:03.887855 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:50:03.900 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:50:03.900254 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:50:14.036 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:50:14.035409 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:50:14.049 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:50:14.049736 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:50:24.162 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:50:24.162056 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:50:24.171 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:50:24.171773 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:50:34.231 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:50:34.231361 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:50:34.287 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:50:34.287720 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:50:44.346 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:50:44.343337 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:50:44.357 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:50:44.353004 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:50:54.426 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:50:54.426640 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:50:54.436 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:50:54.436387 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:51:04.532 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:51:04.532496 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:51:04.544 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:51:04.544803 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:51:14.653 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:51:14.653837 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:51:14.665 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:51:14.665800 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:51:24.744 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:51:24.744638 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:51:24.756 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:51:24.756730 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:51:34.852 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:51:34.851013 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:51:34.862 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:51:34.861796 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:51:44.956 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:51:44.956002 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:51:44.970 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:51:44.969617 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:51:55.083 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:51:55.082788 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:51:55.096 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:51:55.095826 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:52:05.164 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:52:05.164045 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:52:05.179 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:52:05.179704 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:52:15.322 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:52:15.322395 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:52:15.334 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:52:15.334419 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:52:25.399 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:52:25.399408 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:52:25.414 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:52:25.414699 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:52:35.493 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:52:35.493678 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:52:35.551 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:52:35.550716 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:52:45.614 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:52:45.614574 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:52:45.686 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:52:45.686121 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:52:55.749 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:52:55.749620 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:52:55.761 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:52:55.760242 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:53:05.850 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:53:05.850437 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:53:05.924 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:53:05.922766 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:53:16.024 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:53:16.024127 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:53:16.041 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:53:16.041412 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:53:23.514 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:53:23.513832 I | op-mon: parsing mon endpoints: a=10.110.141.123:6789,b=10.98.41.230:6789,c=10.109.47.142:6789 2022-03-09T02:53:24.194 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:53:24.194433 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-09T02:53:24.185+0000 7fa540c5e380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-09T02:53:24.195 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09T02:53:24.185+0000 7fa540c5e380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-09T02:53:24.195 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09T02:53:24.185+0000 7fa540c5e380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-09T02:53:24.195 INFO:tasks.rook.operator.smithi077.stdout:couldn't init storage provider. : exit status 5 2022-03-09T02:53:26.148 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:53:26.148103 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:53:26.160 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:53:26.160338 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:53:36.287 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:53:36.287476 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:53:36.302 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:53:36.302585 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:53:46.371 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:53:46.371458 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:53:46.383 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:53:46.383374 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:53:56.496 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:53:56.496091 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:53:56.509 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:53:56.509158 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:54:06.621 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:54:06.621510 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:54:06.701 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:54:06.632825 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:54:16.730 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:54:16.728394 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:54:16.741 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:54:16.741454 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:54:26.880 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:54:26.880096 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:54:26.892 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:54:26.892868 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:54:37.003 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:54:37.001616 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:54:37.013 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:54:37.013850 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:54:47.133 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:54:47.133955 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:54:47.146 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:54:47.146784 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:54:57.243 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:54:57.243378 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:54:57.257 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:54:57.257712 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:55:07.321 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:55:07.320407 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:55:07.335 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:55:07.334933 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:55:17.460 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:55:17.459506 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:55:17.469 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:55:17.469701 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:55:27.540 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:55:27.540198 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:55:27.552 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:55:27.552231 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:55:37.657 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:55:37.657399 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:55:37.667 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:55:37.667466 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:55:47.732 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:55:47.732439 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:55:47.800 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:55:47.799913 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:55:57.902 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:55:57.901797 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:55:57.920 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:55:57.915376 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:56:08.029 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:56:08.029511 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:56:08.039 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:56:08.039910 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:56:18.130 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:56:18.129346 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:56:18.140 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:56:18.139251 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:56:28.236 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:56:28.236439 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:56:28.301 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:56:28.301502 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:56:38.431 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:56:38.430816 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:56:38.449 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:56:38.449633 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:56:48.534 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:56:48.534655 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:56:48.604 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:56:48.603864 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:56:58.746 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:56:58.746218 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:56:58.757 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:56:58.757997 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:57:08.825 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:57:08.825579 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:57:08.882 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:57:08.882116 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:57:19.012 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:57:19.012068 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:57:19.025 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:57:19.025285 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:57:29.185 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:57:29.185620 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:57:29.197 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:57:29.197260 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:57:39.257 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:57:39.256376 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:57:39.269 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:57:39.269078 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:57:49.372 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:57:49.372591 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:57:49.388 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:57:49.388855 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:57:59.460 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:57:59.460426 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:57:59.472 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:57:59.472195 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:58:09.542 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:58:09.542411 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:58:09.600 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:58:09.600141 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:58:19.703 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:58:19.703086 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:58:19.717 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:58:19.717369 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:58:29.804 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:58:29.803739 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:58:29.816 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:58:29.816337 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:58:39.921 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:58:39.920872 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:58:39.931 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:58:39.931433 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:58:50.033 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:58:50.033485 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:58:50.044 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:58:50.044769 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:59:00.230 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:59:00.229885 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:59:00.242 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:59:00.242489 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:59:10.303 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:59:10.302610 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:59:10.318 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:59:10.318209 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:59:20.386 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:59:20.385931 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:59:20.399 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:59:20.399739 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:59:30.499 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:59:30.499238 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:59:30.514 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:59:30.514284 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:59:40.576 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:59:40.576560 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:59:40.589 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:59:40.589187 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:59:50.707 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:59:50.707466 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:59:50.718 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 02:59:50.718885 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T03:00:00.781 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 03:00:00.781281 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T03:00:00.793 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 03:00:00.793033 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T03:00:10.856 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 03:00:10.856429 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T03:00:10.868 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 03:00:10.868071 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T03:00:20.995 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 03:00:20.995146 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T03:00:21.006 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 03:00:21.006698 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T03:00:31.112 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 03:00:31.112382 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T03:00:31.133 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 03:00:31.133181 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T03:00:41.281 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 03:00:41.281494 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T03:00:41.292 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 03:00:41.292317 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T03:00:51.348 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 03:00:51.348354 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T03:00:51.358 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 03:00:51.358934 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T03:01:01.502 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 03:01:01.502430 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T03:01:01.513 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 03:01:01.514004 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T03:01:11.611 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 03:01:11.611174 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T03:01:11.621 INFO:tasks.rook.operator.smithi077.stdout:2022-03-09 03:01:11.622030 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T05:22:27.144 DEBUG:teuthology.exit:Got signal 15; running 1 handler... 2022-03-09T05:22:27.165 DEBUG:teuthology.task.console_log:Killing console logger for smithi077 2022-03-09T05:22:27.166 DEBUG:teuthology.task.console_log:Killing console logger for smithi106 2022-03-09T05:22:27.166 DEBUG:teuthology.task.console_log:Killing console logger for smithi135 2022-03-09T05:22:27.167 DEBUG:teuthology.exit:Finished running handlers